Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 711787 Details for
Bug 921643
vdsm dosn't refresh size of iscsi luns
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm log
vdsm.log (text/plain), 14.99 MB, created by
Pavel Zhukov
on 2013-03-18 08:09:46 UTC
(
hide
)
Description:
vdsm log
Filename:
MIME Type:
Creator:
Pavel Zhukov
Created:
2013-03-18 08:09:46 UTC
Size:
14.99 MB
patch
obsolete
>Thread-22::DEBUG::2013-03-17 04:01:05,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:01:05,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150135 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:01:07,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:01:07,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289429 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:01:15,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:01:15,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124217 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:01:18,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:01:18,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083396 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:01:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:01:25,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141883 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:01:28,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:01:28,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943156 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:01:35,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:01:35,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868915 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:01:38,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:01:38,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161839 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13739::WARNING::2013-03-17 04:01:40,165::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:01:40,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:01:45,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:01:45,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205929 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:01:48,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:01:48,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846208 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:01:50,173::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:01:50,174::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:01:50,626::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:01:50,632::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:01:50,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:01:55,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:01:55,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259332 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:01:58,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:01:58,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793128 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:02:05,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:02:05,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00698415 s, 586 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:02:08,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:02:08,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160142 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:02:15,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:02:15,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109287 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:02:18,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:02:18,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893028 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:02:25,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:02:25,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870553 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:02:28,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:02:28,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727921 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:02:35,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:02:35,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164943 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:02:38,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:02:38,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169867 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:02:45,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:02:45,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118409 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:02:48,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:02:48,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081496 s, 5.0 MB/s\n'; <rc> = 0 >Thread-13740::WARNING::2013-03-17 04:02:50,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:02:50,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:02:55,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:02:55,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990561 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:02:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:02:58,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763532 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:03:00,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:03:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:03:01,106::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:03:01,112::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:03:01,114::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:03:05,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:03:06,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765679 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:03:08,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:03:08,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906884 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:03:16,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:03:16,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815002 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:03:18,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:03:18,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776467 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:03:26,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:03:26,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880762 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:03:29,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:03:29,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805781 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:03:36,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:03:36,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148695 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:03:39,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:03:39,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774566 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:03:46,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:03:46,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789524 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:03:49,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:03:49,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867913 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:03:56,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:03:56,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181396 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:03:59,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:03:59,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699509 s, 5.9 MB/s\n'; <rc> = 0 >Thread-13741::WARNING::2013-03-17 04:04:01,139::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:04:01,142::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:04:06,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:04:06,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773743 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:09,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085845 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:04:11,144::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:04:11,145::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:04:11,146::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:04:11,538::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:04:11,545::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:04:11,546::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:04:16,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:04:16,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789897 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:19,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:19,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015324 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:04:26,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:04:26,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143275 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:29,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:29,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195712 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:04:36,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:04:36,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101542 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:39,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:39,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011423 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:04:46,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:04:46,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840626 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:49,604::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:04:49,605::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:04:49,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:49,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819676 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:49,635::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819676 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:04:49,636::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:04:49,636::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:04:49,638::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:04:49,639::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:04:49,639::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:04:49,640::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:04:49,640::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:04:49,641::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:04:49,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:50,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:50,047::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:04:50,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:04:50,051::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:04:50,053::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:04:50,054::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:04:50,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:50,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:04:50,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:04:50,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104176 s, 393 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:04:56,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:04:56,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165181 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:05:00,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:05:00,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838505 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:06,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:06,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878754 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:05:10,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:05:10,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799032 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13742::WARNING::2013-03-17 04:05:11,580::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:05:11,583::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:05:16,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:16,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00067179 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:05:20,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:05:21,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138925 s, 295 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:05:21,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:05:21,587::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:05:22,018::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:05:22,025::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:05:22,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:26,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:26,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117777 s, 348 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:05:31,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:05:31,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077429 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:36,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951265 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:05:41,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:05:41,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153105 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:46,669::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:05:46,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:05:46,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:46,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000775943 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:46,697::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000775943 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:05:46,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:05:46,697::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:05:46,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:05:46,701::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:46,702::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:46,702::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:46,703::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:46,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:46,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:47,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:47,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:05:47,142::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:05:47,144::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:05:47,145::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:05:47,146::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:05:47,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:47,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:47,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:47,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153489 s, 267 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:05:51,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:05:51,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166501 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:05:57,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:05:57,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011187 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:06:01,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:06:01,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159704 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:06:07,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:06:07,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164324 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:06:11,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:06:11,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150714 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:06:17,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:06:17,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817665 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:06:21,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:06:21,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082297 s, 5.0 MB/s\n'; <rc> = 0 >Thread-13743::WARNING::2013-03-17 04:06:22,067::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:06:22,075::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:06:27,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:06:27,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840484 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:06:31,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:06:31,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206025 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:06:32,081::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:06:32,082::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:06:32,549::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:06:32,556::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:06:32,557::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:06:37,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:06:37,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00607891 s, 674 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:06:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:06:41,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174255 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:06:47,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:06:47,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123806 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:06:52,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:06:52,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987132 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:06:57,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:06:57,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166336 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:07:02,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:02,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171724 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:07:07,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:07:07,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163331 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:07:12,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:12,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016343 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:07:17,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:07:17,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969469 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:07:22,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:22,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015585 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:07:27,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:07:27,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130079 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:07:32,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:32,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129952 s, 3.2 MB/s\n'; <rc> = 0 >Thread-13744::WARNING::2013-03-17 04:07:32,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:07:32,611::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:07:37,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:07:38,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856791 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:07:42,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:07:42,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:42,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:42,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00921858 s, 444 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:07:43,057::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:07:43,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:07:43,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:07:48,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:07:48,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300384 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:07:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:07:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116148 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:07:58,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:07:58,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993071 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:08:02,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:02,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884143 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:08:08,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:08:08,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904248 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:08:12,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:12,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809054 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:08:18,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:08:18,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:08:22,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:23,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156065 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:08:28,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:08:28,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00797888 s, 513 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:08:33,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:33,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133795 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:08:38,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:08:38,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211777 s, 1.9 MB/s\n'; <rc> = 0 >Thread-13745::WARNING::2013-03-17 04:08:43,102::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:08:43,106::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:08:43,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:43,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822322 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:08:48,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:08:48,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016021 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:08:53,108::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:08:53,109::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:53,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:08:53,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157772 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:08:53,626::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:08:53,631::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:08:53,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:08:58,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:08:58,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102257 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:03,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:03,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805631 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:09:08,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:09:08,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322795 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:13,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906338 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:09:18,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:09:18,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839339 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:23,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:23,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156406 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:09:28,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:09:28,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906922 s, 452 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:33,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:33,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691237 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:09:38,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:09:38,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764464 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:43,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:43,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106014 s, 386 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:09:48,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:09:48,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135798 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13746::WARNING::2013-03-17 04:09:53,659::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:09:53,662::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:09:53,704::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:09:53,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:09:53,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:53,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000862365 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:53,736::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000862365 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:09:53,737::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:09:53,737::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:09:53,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:09:53,740::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:09:53,740::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:09:53,741::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:09:53,741::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:09:53,742::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:09:53,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:54,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:54,223::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:09:54,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:09:54,226::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:09:54,227::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:09:54,228::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:09:54,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:54,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:09:54,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:09:54,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00877157 s, 467 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:09:58,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:09:58,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102271 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:10:03,671::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:10:03,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:10:03,673::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:10:04,086::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:10:04,094::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:10:04,095::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:10:04,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:10:04,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125063 s, 328 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:08,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:08,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078588 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:10:14,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:10:14,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000648219 s, 6.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:18,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:18,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105164 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:10:24,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:10:24,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766943 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:28,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:28,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146302 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:10:34,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:10:34,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803047 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:38,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:38,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161796 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:10:45,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:10:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117013 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:48,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:10:48,983::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:10:48,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:49,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107256 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:49,023::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107256 s, 1.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:10:49,023::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:10:49,023::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:10:49,025::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:10:49,026::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:10:49,026::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:10:49,027::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:10:49,027::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:10:49,028::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:10:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:49,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:49,461::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:10:49,462::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:10:49,465::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:10:49,466::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:10:49,466::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:10:49,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:49,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:49,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:49,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943135 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:10:55,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:10:55,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168307 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:10:59,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:10:59,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001187 s, 3.5 MB/s\n'; <rc> = 0 >Thread-13747::WARNING::2013-03-17 04:11:04,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:11:04,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:11:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:11:05,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168114 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:11:09,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:11:10,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113453 s, 3.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:11:14,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:11:14,140::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:11:14,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:11:14,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:11:14,554::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:11:15,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:11:15,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00870849 s, 470 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:11:20,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:11:20,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787623 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:11:25,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:11:25,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820462 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:11:30,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:11:30,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812879 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:11:35,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:11:35,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154923 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:11:40,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:11:40,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844588 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:11:45,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:11:45,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957883 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:11:50,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:11:50,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151502 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:11:55,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:11:55,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884213 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:12:00,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:12:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781538 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:12:05,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:12:05,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762374 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:12:10,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:12:10,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876586 s, 4.7 MB/s\n'; <rc> = 0 >Thread-13748::WARNING::2013-03-17 04:12:14,580::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:12:14,583::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:12:15,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:12:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113943 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:12:20,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:12:20,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834706 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:12:24,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:12:24,587::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:12:25,004::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:12:25,011::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:12:25,012::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:12:25,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:12:25,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124462 s, 329 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:12:30,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:12:30,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866394 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:12:35,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:12:35,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796822 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:12:40,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:12:40,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145983 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:12:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:12:45,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791024 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:12:50,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:12:50,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881904 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:12:56,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:12:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142924 s, 287 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:13:00,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:13:00,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164918 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:13:06,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:13:06,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180319 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:13:10,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:13:10,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164205 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:13:16,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:13:16,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204267 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:13:20,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:13:20,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150678 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13749::WARNING::2013-03-17 04:13:25,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:13:25,045::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:13:26,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:13:26,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898275 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:13:30,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:13:30,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231114 s, 1.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:13:35,049::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:13:35,050::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:13:35,469::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:13:35,476::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:13:35,477::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:13:36,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:13:36,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318615 s, 129 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:13:40,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:13:40,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000681067 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:13:46,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:13:46,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141691 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:13:50,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:13:50,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016087 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:13:56,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:13:56,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774607 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:14:00,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:14:00,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787392 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:06,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:06,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088529 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:14:10,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:14:10,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124068 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:16,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:16,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848777 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:14:20,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:14:20,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149196 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:26,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:26,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878367 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:14:30,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:14:30,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156641 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13750::WARNING::2013-03-17 04:14:35,507::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:14:35,510::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:14:36,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:36,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764119 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:14:40,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:14:40,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788868 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:14:45,512::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:14:45,513::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:14:45,944::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:14:45,952::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:14:45,954::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:46,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:46,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010252 s, 400 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:14:50,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:14:50,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919037 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:57,015::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:14:57,017::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:14:57,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:57,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000880181 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:57,051::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000880181 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:14:57,051::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:14:57,051::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:14:57,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:14:57,054::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:57,055::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:57,055::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:57,055::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:57,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:57,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:57,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:57,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:14:57,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:14:57,481::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:14:57,483::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:14:57,484::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:14:57,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:57,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:14:57,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:14:57,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132171 s, 310 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:00,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:00,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717816 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:15:08,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:15:08,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851593 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:11,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:11,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138581 s, 296 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:15:18,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:15:18,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765069 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:21,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:21,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091068 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:15:28,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:15:28,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142549 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:31,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:31,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106855 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:15:38,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:15:38,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160225 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:41,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:41,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160024 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13751::WARNING::2013-03-17 04:15:45,981::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:15:45,985::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:15:48,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:15:48,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858002 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:51,239::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:15:51,241::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:15:51,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:51,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0221263 s, 92.6 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:51,302::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0221263 s, 92.6 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:15:51,302::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:15:51,302::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:15:51,305::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:15:51,306::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:15:51,307::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:15:51,307::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:15:51,307::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:15:51,308::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:15:51,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:51,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:51,726::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:15:51,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:15:51,735::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:15:51,736::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:15:51,737::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:15:51,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:52,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:15:52,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:15:52,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171976 s, 238 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:15:55,986::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:15:55,987::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:15:55,988::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:15:56,419::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:15:56,425::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:15:56,427::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:15:58,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:15:58,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139506 s, 294 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:16:02,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:16:02,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851534 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:16:08,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:16:08,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764694 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:16:12,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:16:12,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808393 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:16:18,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:16:18,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793406 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:16:22,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:16:22,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771134 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:16:28,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:16:28,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765498 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:16:32,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:16:32,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795232 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:16:38,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:16:38,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748681 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:16:42,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:16:42,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118422 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:16:48,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:16:48,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101941 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:16:52,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:16:52,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151807 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13752::WARNING::2013-03-17 04:16:56,453::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:16:56,466::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:16:58,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:16:58,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706602 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:17:02,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:17:02,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154846 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:17:06,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:17:06,475::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:17:06,948::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:17:06,956::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:17:06,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:17:08,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:17:08,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104965 s, 390 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:17:12,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:17:12,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155805 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:17:18,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:17:19,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083257 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:17:22,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:17:22,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958058 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:17:29,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:17:29,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154221 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:17:32,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:17:32,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001222 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:17:39,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:17:39,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834324 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:17:42,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:17:42,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078927 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:17:49,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:17:49,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145032 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:17:52,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:17:52,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153244 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:17:59,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:17:59,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107673 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:18:02,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:18:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164142 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13753::WARNING::2013-03-17 04:18:06,985::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:18:06,993::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:18:09,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:18:09,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162599 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:18:12,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:18:12,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841607 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:18:16,995::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:18:16,997::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:18:17,432::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:18:17,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:18:17,441::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:18:19,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:18:19,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00351068 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:18:22,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:18:22,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154804 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:18:29,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:18:29,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069531 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:18:32,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:18:32,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142619 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:18:39,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:18:39,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861437 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:18:42,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:18:42,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173544 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:18:49,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:18:49,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157678 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:18:52,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:18:52,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156994 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:18:59,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:18:59,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808439 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:19:02,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:19:02,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143034 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:19:09,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:19:09,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083996 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:19:12,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:19:13,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141629 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13754::WARNING::2013-03-17 04:19:17,477::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:19:17,482::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:19:19,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:19:19,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965632 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:19:23,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:19:23,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825389 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:19:27,490::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:19:27,491::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:19:27,904::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:19:27,911::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:19:27,912::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:19:29,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:19:30,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124694 s, 328 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:19:33,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:19:33,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151338 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:19:40,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:19:40,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826797 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:19:43,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:19:43,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155408 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:19:50,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:19:50,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161043 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:19:53,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:19:53,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872229 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:00,227::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:20:00,229::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:20:00,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:00,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000828647 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:00,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000828647 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:20:00,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:20:00,259::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:20:00,261::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:20:00,262::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:00,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:00,263::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:00,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:00,264::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:00,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:00,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:00,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:00,675::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:20:00,678::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:20:00,679::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:20:00,679::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:20:00,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:01,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:01,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:01,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179323 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:03,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:03,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836409 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:11,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:11,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080803 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:13,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:13,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759148 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:21,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:21,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166941 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:23,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:23,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152461 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13755::WARNING::2013-03-17 04:20:27,939::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:20:27,942::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:20:31,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:31,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749171 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:33,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:33,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016719 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:20:37,948::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:20:37,949::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:20:38,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:20:38,415::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:20:38,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:20:41,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:41,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00889916 s, 460 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:43,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:43,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211167 s, 194 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:20:51,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:20:51,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752935 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:53,427::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:20:53,429::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:20:53,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:53,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000750872 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:53,457::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000750872 s, 2.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:20:53,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:20:53,458::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:20:53,459::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:20:53,464::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:20:53,465::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:20:53,465::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:20:53,466::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:20:53,466::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:20:53,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:53,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:53,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:20:53,901::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:20:53,906::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:20:53,908::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:20:53,909::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:20:53,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:54,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:20:54,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:20:54,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778017 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:21:01,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:21:01,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789532 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:21:04,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:21:04,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880036 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:21:11,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:21:11,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785105 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:21:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:21:14,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079392 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:21:21,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:21:21,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140721 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:21:24,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:21:24,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143537 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:21:31,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:21:31,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140278 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:21:34,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:21:34,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778864 s, 5.3 MB/s\n'; <rc> = 0 >Thread-13756::WARNING::2013-03-17 04:21:38,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:21:38,452::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:21:41,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:21:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832738 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:21:44,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:21:44,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831497 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:21:48,460::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:21:48,461::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:21:48,462::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:21:48,917::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:21:48,924::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:21:48,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:21:51,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:21:52,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00985833 s, 415 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:21:54,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:21:54,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069753 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:22:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:22:02,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180891 s, 226 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:22:04,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:22:04,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893408 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:22:12,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:22:12,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786524 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:22:14,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:22:14,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154196 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:22:22,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:22:22,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110686 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:22:24,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:22:24,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103389 s, 396 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:22:32,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:22:32,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154412 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:22:34,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:22:34,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825377 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:22:42,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:22:42,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182695 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:22:44,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:22:44,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157007 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13757::WARNING::2013-03-17 04:22:48,967::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:22:48,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:22:52,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:22:52,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027499 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:22:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:22:54,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720306 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:22:58,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:22:58,975::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:22:59,492::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:22:59,499::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:22:59,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:23:02,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:23:02,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133835 s, 306 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:23:04,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:23:04,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159993 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:23:12,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:23:12,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163418 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:23:14,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:23:14,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081694 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:23:22,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:23:22,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369184 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:23:24,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:23:24,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888577 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:23:32,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:23:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154474 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:23:34,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:23:34,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00238152 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:23:42,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:23:42,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007987 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:23:44,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:23:45,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847595 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:23:52,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:23:52,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0283431 s, 145 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:23:55,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:23:55,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177929 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13758::WARNING::2013-03-17 04:23:59,533::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:23:59,537::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:24:03,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:24:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790572 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:24:05,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:24:05,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00830474 s, 493 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:24:09,545::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:24:09,547::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:24:10,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:24:10,072::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:24:10,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:24:13,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:24:13,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161805 s, 253 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:24:15,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:24:15,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.035832 s, 114 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:24:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:24:23,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762609 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:24:25,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:24:25,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967734 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:24:33,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:24:33,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774338 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:24:35,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:24:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085706 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:24:43,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:24:43,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016359 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:24:45,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:24:45,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172017 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:24:53,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:24:53,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148675 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:24:55,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:24:55,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153789 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:03,504::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:25:03,505::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:25:03,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:03,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000760176 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:03,545::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000760176 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:25:03,546::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:25:03,546::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:25:03,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:25:03,550::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:03,551::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:03,551::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:03,553::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:03,553::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:03,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:03,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:04,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:04,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:25:04,007::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:25:04,008::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:25:04,009::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:25:04,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:04,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:04,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:04,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101622 s, 403 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:05,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:05,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183608 s, 223 kB/s\n'; <rc> = 0 >Thread-13759::WARNING::2013-03-17 04:25:10,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:25:10,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:25:14,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:14,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00291446 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:15,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:15,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167137 s, 245 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:25:20,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:25:20,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:25:20,611::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:25:20,618::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:25:20,619::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:25:24,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:24,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0260636 s, 157 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:25,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:25,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167028 s, 245 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:34,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:34,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154779 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:35,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:35,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00531641 s, 770 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:44,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:44,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784271 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:45,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:45,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159558 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:25:54,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:25:54,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161694 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:55,627::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:25:55,629::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:25:55,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:55,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000880962 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:55,671::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000880962 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:25:55,671::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:25:55,671::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:25:55,673::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:25:55,674::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:25:55,674::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:25:55,675::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:25:55,675::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:25:55,676::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:25:55,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:56,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:56,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:25:56,126::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:25:56,128::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:25:56,129::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:25:56,130::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:25:56,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:56,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:25:56,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:25:56,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126071 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:26:04,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:26:04,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854461 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:26:06,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:26:06,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314746 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:26:14,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:26:14,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831249 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:26:16,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:26:16,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141554 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13760::WARNING::2013-03-17 04:26:20,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:26:20,666::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:26:25,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:26:25,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861434 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:26:26,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:26:26,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167712 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:26:30,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:26:30,670::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:26:31,200::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:26:31,206::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:26:31,208::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:26:35,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:26:35,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00996745 s, 411 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:26:36,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:26:36,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101084 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:26:45,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:26:45,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762126 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:26:46,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:26:46,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755375 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:26:55,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:26:55,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014547 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:26:56,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:26:56,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770018 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:27:05,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:27:05,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156372 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:27:06,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:27:06,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106681 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:27:15,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:27:15,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154461 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:27:16,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:27:16,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147661 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:27:25,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:27:25,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123435 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:27:26,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:27:26,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156665 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13761::WARNING::2013-03-17 04:27:31,238::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:27:31,245::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:27:35,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:27:35,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805967 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:27:37,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:27:37,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803138 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:27:41,247::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:27:41,249::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:27:41,249::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:27:41,720::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:27:41,726::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:27:41,728::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:27:45,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:27:45,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931784 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:27:47,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:27:47,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829248 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:27:55,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:27:55,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810261 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:27:57,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:27:57,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867711 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:28:05,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:28:05,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745742 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:28:07,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:28:07,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125304 s, 327 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:28:15,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:28:15,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805323 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:28:17,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:28:17,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755386 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:28:25,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:28:25,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924782 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:28:27,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:28:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157577 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:28:36,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:28:36,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737515 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:28:37,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:28:37,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849995 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13762::WARNING::2013-03-17 04:28:41,754::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:28:41,758::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:28:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:28:46,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773813 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:28:47,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:28:47,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205187 s, 200 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:28:51,760::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:28:51,761::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:28:52,218::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:28:52,225::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:28:52,226::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:28:56,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:28:56,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014455 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:28:57,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:28:57,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719374 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:29:06,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:29:06,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792045 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:29:07,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:29:07,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841783 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:29:16,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:29:16,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765766 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:29:17,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:29:17,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195561 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:29:26,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:29:26,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001532 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:29:27,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:29:27,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180861 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:29:36,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:29:36,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892302 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:29:37,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:29:37,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970595 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:29:46,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:29:46,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013495 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:29:47,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:29:47,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150132 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13763::WARNING::2013-03-17 04:29:52,253::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:29:52,257::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:29:56,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:29:56,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892789 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:29:57,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:29:57,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153225 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:30:02,259::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:30:02,260::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:30:02,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:30:02,732::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:30:02,733::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:06,672::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:30:06,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:30:06,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:06,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000648309 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:06,708::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000648309 s, 3.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:30:06,708::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:30:06,708::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:30:06,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:30:06,711::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:06,712::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:06,712::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:06,712::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:06,713::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:07,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:07,164::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:30:07,165::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:30:07,167::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:30:07,168::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:30:07,169::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:30:07,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:07,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115135 s, 356 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:07,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:07,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163517 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:17,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:17,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289652 s, 141 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:17,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:17,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0403156 s, 102 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:27,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:28,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00685884 s, 597 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:28,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:28,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234049 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:38,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:38,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847187 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:38,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:38,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160215 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:48,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:48,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103916 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:30:48,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:48,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689263 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:58,124::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:30:58,126::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:30:58,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:58,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107514 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:58,161::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107514 s, 1.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:30:58,161::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:30:58,162::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:30:58,163::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:30:58,164::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:30:58,165::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:30:58,166::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:30:58,167::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:30:58,167::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:30:58,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:58,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:30:58,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00604239 s, 678 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:58,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:58,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:30:58,652::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:30:58,655::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:30:58,656::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:30:58,656::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:30:58,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:59,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:30:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:30:59,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171827 s, 238 kB/s\n'; <rc> = 0 >Thread-13764::WARNING::2013-03-17 04:31:02,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:31:02,771::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:31:08,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:31:08,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905618 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:31:09,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:31:09,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778991 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:31:12,773::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:31:12,774::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:31:13,229::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:31:13,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:31:13,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:31:18,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:31:18,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695039 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:31:19,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:31:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138729 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:31:28,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:31:28,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823113 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:31:29,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:31:29,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955786 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:31:38,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:31:38,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738347 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:31:39,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:31:39,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167653 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:31:48,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:31:48,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159397 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:31:49,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:31:49,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195311 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:31:58,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:31:58,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148725 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:31:59,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:31:59,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919484 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:32:08,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:32:08,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012098 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:32:09,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:32:09,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781418 s, 5.2 MB/s\n'; <rc> = 0 >Thread-13765::WARNING::2013-03-17 04:32:13,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:32:13,264::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:32:18,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:32:18,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165674 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:32:19,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:32:19,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798083 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:32:23,266::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:32:23,267::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:32:23,825::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:32:23,832::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:32:23,833::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:32:28,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:32:29,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0270155 s, 152 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:32:29,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:32:29,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755479 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:32:39,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:32:39,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956056 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:32:39,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:32:39,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882676 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:32:49,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:32:49,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842402 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:32:49,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:32:49,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846488 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:32:59,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:32:59,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140595 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:32:59,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:32:59,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776564 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:33:09,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:33:09,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862993 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:33:09,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:33:09,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102371 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:33:19,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:33:19,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177753 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:33:19,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:33:19,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150895 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13766::WARNING::2013-03-17 04:33:23,859::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:33:23,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:33:29,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:33:29,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719233 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:33:29,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:33:29,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932395 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:33:33,864::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:33:33,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:33:33,867::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:33:34,325::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:33:34,331::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:33:34,333::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:33:39,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:33:39,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165622 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:33:39,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:33:39,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746861 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:33:49,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:33:49,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00305025 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:33:49,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:33:49,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166781 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:33:59,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:33:59,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00332577 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:33:59,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:33:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156411 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:34:09,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:34:09,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179994 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:34:09,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:34:09,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073343 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:34:19,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:34:19,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752259 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:34:19,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:34:20,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017179 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:34:30,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:34:30,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:34:30,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011353 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:34:30,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115339 s, 3.6 MB/s\n'; <rc> = 0 >Thread-13767::WARNING::2013-03-17 04:34:34,359::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:34:34,363::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:34:40,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:34:40,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909307 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:34:40,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:34:40,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756458 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:34:44,365::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:34:44,366::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:34:44,805::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:34:44,811::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:34:44,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:34:50,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:34:50,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758266 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:34:50,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:34:50,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862714 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:35:00,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:35:00,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149158 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:00,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:00,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801476 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:35:10,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:35:10,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744988 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:10,423::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:35:10,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:35:10,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:10,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135822 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:10,472::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135822 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:35:10,472::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:35:10,473::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:35:10,474::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:35:10,475::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:35:10,476::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:35:10,476::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:35:10,477::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:35:10,477::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:35:10,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:10,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:10,916::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:35:10,917::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:35:10,920::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:35:10,921::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:35:10,921::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:35:10,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:11,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:11,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00657439 s, 623 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:35:20,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:35:20,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070917 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:21,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:21,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111755 s, 367 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:35:30,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:35:30,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153527 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:31,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:31,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172935 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:35:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:35:40,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00261049 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:41,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:41,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151238 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13768::WARNING::2013-03-17 04:35:44,839::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:35:44,842::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:35:50,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:35:50,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766242 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:35:51,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:35:51,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772731 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:35:54,844::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:35:54,846::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:35:55,305::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:35:55,312::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:35:55,313::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:00,554::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:36:00,556::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:36:00,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00108615 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:00,595::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00108615 s, 1.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:36:00,596::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:36:00,596::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:36:00,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:36:00,598::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:00,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:00,600::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:00,600::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:00,600::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:00,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:01,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:01,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:36:01,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:36:01,039::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:36:01,046::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:36:01,046::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:36:01,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:01,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:01,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:01,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0389538 s, 105 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:36:01,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:36:01,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00877542 s, 467 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:11,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:11,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00326238 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:36:11,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:36:11,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00296738 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:21,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:21,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159175 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:36:21,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:36:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777451 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:31,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:31,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738673 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:36:31,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:36:31,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091579 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:41,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:41,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897029 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:36:42,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:36:42,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015174 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:36:51,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:36:51,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120473 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:36:52,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:36:52,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989383 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13769::WARNING::2013-03-17 04:36:55,339::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:36:55,345::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:37:01,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:37:01,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978648 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:37:02,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:37:02,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909809 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:37:05,349::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:37:05,350::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:37:05,794::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:37:05,800::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:37:05,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:37:11,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:37:11,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211713 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:37:12,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:37:12,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927266 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:37:21,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:37:21,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860367 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:37:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:37:22,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789752 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:37:31,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:37:31,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803088 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:37:32,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:37:32,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190071 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:37:41,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:37:41,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734364 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:37:42,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:37:42,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156472 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:37:51,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:37:52,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373678 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:37:52,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:37:52,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153564 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:38:02,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:38:02,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944041 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:38:02,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:38:02,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276086 s, 1.5 MB/s\n'; <rc> = 0 >Thread-13770::WARNING::2013-03-17 04:38:05,834::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:38:05,838::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:38:12,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:38:12,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091984 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:38:12,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:38:12,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780474 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:38:15,841::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:38:15,843::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:38:16,292::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:38:16,300::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:38:16,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:38:22,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:38:22,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150605 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:38:22,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:38:22,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101229 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:38:32,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:38:32,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132712 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:38:32,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:38:32,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016343 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:38:42,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:38:42,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158658 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:38:42,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:38:43,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244195 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:38:52,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:38:52,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897345 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:38:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:38:53,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800763 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:39:02,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:39:02,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145754 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:39:03,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:39:03,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159616 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:39:12,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:39:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150083 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:39:13,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:39:13,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155942 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13771::WARNING::2013-03-17 04:39:16,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:39:16,331::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:39:22,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:39:22,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867931 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:39:23,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:39:23,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793483 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:39:26,332::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:39:26,333::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:39:26,334::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:39:26,852::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:39:26,858::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:39:26,860::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:39:32,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:39:32,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841982 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:39:33,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:39:33,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811922 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:39:42,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:39:42,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861511 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:39:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:39:43,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173038 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:39:52,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:39:52,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153693 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:39:53,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:39:53,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775245 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:40:02,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:40:02,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752833 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:03,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:03,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016379 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:40:12,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:40:12,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164795 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:13,657::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:40:13,658::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:40:13,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:13,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137542 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:13,702::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137542 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:40:13,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:40:13,703::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:40:13,705::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:40:13,706::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:40:13,706::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:40:13,707::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:40:13,707::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:40:13,708::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:40:13,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:14,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:14,137::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:40:14,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:40:14,141::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:40:14,142::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:40:14,142::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:40:14,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:14,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:14,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:14,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133327 s, 307 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:40:22,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:40:22,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00348758 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:24,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:24,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231186 s, 177 kB/s\n'; <rc> = 0 >Thread-13772::WARNING::2013-03-17 04:40:26,885::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:40:26,888::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:40:32,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:40:32,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081397 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:34,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:34,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170875 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:40:36,891::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:40:36,893::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:40:37,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:40:37,337::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:40:37,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:40:42,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:40:42,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00381247 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:44,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:44,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008675 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:40:52,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:40:52,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166379 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:40:54,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:40:54,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159047 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:02,816::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:41:02,817::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:41:02,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:02,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138534 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:02,873::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138534 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:41:02,874::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:41:02,874::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:41:02,878::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:41:02,879::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:02,880::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:02,880::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:02,881::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:02,881::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:02,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:03,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:03,321::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:03,322::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:41:03,325::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:41:03,326::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:41:03,326::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:41:03,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:03,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:03,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:03,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101745 s, 403 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:41:04,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:41:05,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829282 s, 494 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:13,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:13,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834389 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:41:15,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:41:15,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176242 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:23,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:23,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152717 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:41:25,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:41:25,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100718 s, 407 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:41:33,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:33,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142794 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:41:35,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:41:35,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739262 s, 5.5 MB/s\n'; <rc> = 0 >Thread-13773::WARNING::2013-03-17 04:41:37,367::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:41:37,372::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:41:43,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:43,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901255 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:41:45,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:41:45,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009142 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:41:47,376::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:41:47,377::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:41:47,988::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:41:47,996::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:41:47,997::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:41:53,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:41:53,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088526 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:41:55,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:41:55,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857084 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:42:03,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:42:04,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146047 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:42:05,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:42:05,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147946 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:42:14,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:42:14,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:42:15,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:42:15,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808968 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:42:24,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:42:24,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159304 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:42:25,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:42:25,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188798 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:42:34,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:42:34,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152172 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:42:35,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:42:35,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775988 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:42:44,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:42:44,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149596 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:42:45,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:42:45,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162776 s, 252 kB/s\n'; <rc> = 0 >Thread-13774::WARNING::2013-03-17 04:42:48,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:42:48,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:42:54,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:42:54,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917734 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:42:55,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:42:55,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807764 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:42:58,066::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:42:58,067::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:42:58,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:42:58,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:42:58,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:43:04,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:43:04,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014825 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:43:05,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:43:05,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834176 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:43:14,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:43:14,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178783 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:43:16,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:43:16,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862692 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:43:24,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:43:24,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768652 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:43:26,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:43:26,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145852 s, 281 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:43:34,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:43:34,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134551 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:43:36,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:43:36,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773438 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:43:44,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:43:44,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911673 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:43:46,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:43:46,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153972 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:43:54,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:43:54,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784909 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:43:56,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:43:56,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132528 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13775::WARNING::2013-03-17 04:43:58,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:43:58,594::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:44:04,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:44:04,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835483 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:44:06,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:44:06,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710649 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:44:08,596::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:44:08,597::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:44:09,040::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:44:09,046::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:44:09,047::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:44:14,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:44:14,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081341 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:44:16,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:44:16,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154934 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:44:24,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:44:24,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822179 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:44:26,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:44:26,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099408 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:44:34,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:44:34,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224153 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:44:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:44:36,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183416 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:44:44,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:44:44,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922018 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:44:46,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:44:46,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829841 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:44:54,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:44:54,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173241 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:44:56,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:44:56,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110629 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:45:04,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:45:04,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00917964 s, 446 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:06,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:06,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124055 s, 3.3 MB/s\n'; <rc> = 0 >Thread-13776::WARNING::2013-03-17 04:45:09,074::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:45:09,078::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:45:14,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:45:14,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154279 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:17,035::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:45:17,037::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:45:17,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:17,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00104436 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:17,075::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00104436 s, 2.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:45:17,075::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:45:17,076::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:45:17,078::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:45:17,079::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:45:17,080::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:45:17,081::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:45:17,081::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:45:17,082::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:45:17,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:17,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:17,499::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:45:17,500::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:45:17,503::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:45:17,504::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:45:17,505::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:45:17,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:17,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:17,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:17,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189602 s, 216 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:45:19,082::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:45:19,084::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:45:19,085::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:45:19,493::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:45:19,499::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:45:19,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:45:24,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:45:24,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0077207 s, 531 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:27,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:28,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117269 s, 349 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:45:34,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:45:34,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014644 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:38,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:38,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916421 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:45:44,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:45:44,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150822 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:48,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:48,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859078 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:45:54,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:45:54,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161547 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:45:58,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:45:58,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159695 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:04,997::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:46:04,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:46:04,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:05,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013691 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:05,046::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013691 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:46:05,047::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:46:05,047::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:46:05,051::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:46:05,053::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:05,053::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:05,054::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:05,054::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:05,054::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:05,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:05,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:05,462::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:05,463::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:46:05,465::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:46:05,466::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:46:05,467::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:46:05,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:05,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:05,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:05,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746001 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:46:08,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:46:08,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00816758 s, 501 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:15,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:15,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144144 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:46:18,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:46:18,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00441303 s, 928 kB/s\n'; <rc> = 0 >Thread-13777::WARNING::2013-03-17 04:46:19,545::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:46:19,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:46:25,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:25,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821836 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:46:28,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:46:28,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00521841 s, 785 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:46:29,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:46:29,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:46:30,012::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:46:30,019::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:46:30,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:46:35,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:35,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160957 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:46:38,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:46:38,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156413 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:45,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:46,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177675 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:46:48,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:46:48,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082321 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:46:56,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:46:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687926 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:46:58,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:46:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805319 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:47:06,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:47:06,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181934 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:47:08,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:47:08,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748273 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:47:16,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:47:16,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769317 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:47:18,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:47:18,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851691 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:47:26,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:47:26,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129385 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:47:28,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:47:29,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00844909 s, 485 kB/s\n'; <rc> = 0 >Thread-13778::WARNING::2013-03-17 04:47:30,050::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:47:30,054::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:47:36,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:47:36,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155982 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:47:39,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:47:39,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863558 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:47:40,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:47:40,057::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:47:40,490::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:47:40,498::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:47:40,499::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:47:46,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:47:46,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934371 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:47:49,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:47:49,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154613 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:47:56,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:47:56,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156424 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:47:59,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:47:59,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915401 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:48:06,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:48:06,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837633 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:48:09,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:48:09,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093918 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:48:16,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:48:16,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182235 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:48:19,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:48:19,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101738 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:48:26,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:48:26,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139448 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:48:29,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:48:29,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181735 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:48:36,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:48:36,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154499 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:48:39,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:48:39,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178996 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13779::WARNING::2013-03-17 04:48:40,529::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:48:40,536::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:48:46,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:48:46,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780427 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:48:50,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:48:50,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819264 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:48:50,539::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:48:50,540::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:48:50,943::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:48:50,950::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:48:50,951::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:48:56,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:48:56,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888862 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:49:00,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:49:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971231 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:49:06,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:49:06,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138322 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:49:10,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:49:10,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833223 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:49:16,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:49:16,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173239 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:49:20,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:49:20,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802848 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:49:26,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:49:26,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145511 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:49:30,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:49:30,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814222 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:49:36,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:49:36,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120337 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:49:40,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:49:40,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769156 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:49:46,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:49:46,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175777 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:49:50,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:49:50,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803231 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13780::WARNING::2013-03-17 04:49:50,983::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:49:50,986::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 04:49:56,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:49:56,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140743 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:00,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:00,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159722 s, 256 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:50:00,988::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:50:00,989::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:50:01,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:50:01,684::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:50:01,686::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:50:06,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:50:06,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00298023 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:10,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:10,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211663 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:50:16,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:50:16,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788062 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:20,951::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:50:20,953::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:50:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136898 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:20,998::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136898 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:50:20,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:50:20,999::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:50:21,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:50:21,001::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:50:21,002::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:50:21,003::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:50:21,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:50:21,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:50:21,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:21,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:21,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:50:21,443::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:50:21,445::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:50:21,447::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:50:21,447::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:50:21,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:21,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:21,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:21,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109392 s, 374 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:50:26,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:50:27,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809056 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:31,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:31,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168358 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:50:37,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:50:37,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00675381 s, 606 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:42,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:42,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153171 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:50:47,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:50:47,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168156 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:50:52,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:50:52,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826552 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:50:57,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:50:57,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150196 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13781::WARNING::2013-03-17 04:51:01,712::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:51:01,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:51:02,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:51:02,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839476 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:07,177::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:51:07,179::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:51:07,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:07,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000871293 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:07,216::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000871293 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:51:07,217::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:51:07,217::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:51:07,219::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:51:07,220::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:51:07,221::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:51:07,222::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:51:07,222::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:51:07,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:51:07,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:07,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:07,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:51:07,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:51:07,652::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:51:07,653::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:51:07,654::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:51:07,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:08,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:08,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:08,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187662 s, 218 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:51:11,718::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:51:11,719::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:51:11,720::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:51:12,200::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:51:12,214::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:51:12,216::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:51:12,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:51:12,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015615 s, 262 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:18,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:18,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964298 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:51:22,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:51:22,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873168 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:28,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:28,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777576 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:51:32,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:51:32,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855733 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:38,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:38,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824268 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:51:42,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:51:42,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757022 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:48,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:48,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149842 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:51:52,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:51:52,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013356 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:51:58,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:51:58,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818729 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:52:02,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:52:02,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155096 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:52:08,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:52:08,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848364 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13782::WARNING::2013-03-17 04:52:12,241::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:52:12,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:52:12,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:52:12,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0071172 s, 576 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:52:18,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:52:18,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836641 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:52:22,249::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:52:22,250::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:52:22,709::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:52:22,715::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:52:22,716::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:52:22,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:52:22,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127766 s, 321 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:52:28,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:52:28,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147767 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:52:32,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:52:32,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176459 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:52:38,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:52:38,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874523 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:52:43,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:52:43,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185872 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:52:48,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:52:48,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770746 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:52:53,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:52:53,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847677 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:52:58,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:52:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299872 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:53:03,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:53:03,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821639 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:53:08,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:53:08,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153036 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:53:13,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:53:13,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851866 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:53:18,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:53:18,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148318 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13783::WARNING::2013-03-17 04:53:22,742::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:53:22,746::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:53:23,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:53:23,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177967 s, 230 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:53:28,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:53:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817796 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:53:32,748::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:53:32,750::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:53:33,167::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:53:33,174::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:53:33,175::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:53:33,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:53:33,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00952789 s, 430 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:53:38,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:53:38,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930554 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:53:43,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:53:43,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148268 s, 276 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:53:48,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:53:48,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815161 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:53:53,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:53:53,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101031 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:53:58,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:53:58,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864636 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:54:03,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:54:03,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151922 s, 270 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:54:08,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:54:08,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910679 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:54:13,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:54:13,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276622 s, 148 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:54:18,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:54:18,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00562603 s, 728 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:54:23,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:54:23,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017701 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:54:28,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:54:28,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938203 s, 4.4 MB/s\n'; <rc> = 0 >Thread-13784::WARNING::2013-03-17 04:54:33,200::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:54:33,203::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:54:33,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:54:33,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908538 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:54:38,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:54:38,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747818 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:54:43,206::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:54:43,207::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:54:43,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:54:43,618::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:54:43,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:54:44,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:54:44,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019712 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:54:48,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:54:49,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014116 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:54:54,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:54:54,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844344 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:54:59,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:54:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937252 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:04,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:04,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125903 s, 325 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:55:09,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:55:09,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140237 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:14,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:14,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708771 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:55:19,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:55:19,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179157 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:24,337::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 04:55:24,339::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:55:24,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:24,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0106251 s, 193 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:24,382::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0106251 s, 193 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 04:55:24,383::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 04:55:24,384::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 04:55:24,387::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 04:55:24,388::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:24,389::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:24,389::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:24,390::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:24,390::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:24,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:24,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:24,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:24,869::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 04:55:24,871::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 04:55:24,873::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 04:55:24,873::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 04:55:24,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:25,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:25,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:25,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00460565 s, 889 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:55:29,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:55:29,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00605886 s, 676 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:55:35,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:35,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200462 s, 204 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:55:39,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:55:39,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106662 s, 3.8 MB/s\n'; <rc> = 0 >Thread-13785::WARNING::2013-03-17 04:55:43,647::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:55:43,651::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:55:45,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:45,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129419 s, 316 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:55:49,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:55:49,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191962 s, 2.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:55:53,653::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:55:53,655::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:55:54,053::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:55:54,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:55:54,061::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:55:55,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:55:55,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013139 s, 312 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:55:59,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:55:59,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0249148 s, 164 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:56:05,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:56:05,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898374 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:09,469::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 04:56:09,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:56:09,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:09,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000930401 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:09,510::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000930401 s, 2.2 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 04:56:09,511::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 04:56:09,511::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 04:56:09,513::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 04:56:09,514::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:56:09,514::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:56:09,515::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:56:09,515::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:56:09,515::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 04:56:09,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:09,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:09,939::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 04:56:09,939::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 04:56:09,942::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 04:56:09,943::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 04:56:09,944::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 04:56:09,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:10,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:10,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:10,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00541413 s, 757 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:56:15,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:56:15,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00780351 s, 525 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:20,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:20,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984555 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:56:25,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:56:25,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120386 s, 340 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:30,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:30,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176949 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:56:35,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:56:35,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155159 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:40,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:40,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087887 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:56:45,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:56:46,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724641 s, 565 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:56:50,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:56:50,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797637 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13786::WARNING::2013-03-17 04:56:54,089::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:56:54,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:56:56,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:56:56,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878388 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:57:00,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:57:00,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814364 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:57:04,095::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 04:57:04,096::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:57:04,097::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:57:04,566::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:57:04,572::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:57:04,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:57:06,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:57:06,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117439 s, 349 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:57:10,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:57:10,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015909 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:57:16,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:57:16,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125632 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:57:20,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:57:20,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000674625 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:57:26,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:57:26,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735107 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:57:30,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:57:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00845753 s, 484 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:57:36,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:57:36,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153667 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:57:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:57:40,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152219 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:57:46,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:57:46,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00392708 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:57:50,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:57:50,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917234 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:57:56,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:57:56,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168473 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:58:00,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:58:00,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180434 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13787::WARNING::2013-03-17 04:58:04,599::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:58:04,603::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:58:06,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:58:06,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900969 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:58:10,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:58:10,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835629 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:58:14,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:58:14,607::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:58:15,027::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:58:15,034::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:58:15,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:58:16,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:58:16,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098774 s, 415 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:58:20,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:58:20,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140115 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:58:26,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:58:26,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120408 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:58:30,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:58:30,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846984 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:58:36,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:58:36,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143848 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:58:41,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:58:41,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186136 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:58:46,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:58:46,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178557 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:58:51,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:58:51,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174706 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:58:57,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:58:57,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817237 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:59:01,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:59:01,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148932 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:59:07,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:59:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120979 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:59:11,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:59:11,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154248 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13788::WARNING::2013-03-17 04:59:15,078::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 04:59:15,084::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 04:59:17,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:59:17,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787039 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:59:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:59:21,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092803 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 04:59:25,097::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 04:59:25,099::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 04:59:25,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 04:59:25,509::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 04:59:25,510::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 04:59:27,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:59:27,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139583 s, 293 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:59:31,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:59:31,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794601 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:59:37,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:59:37,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859901 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:59:41,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:59:41,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013935 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:59:47,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:59:47,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770933 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 04:59:51,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 04:59:51,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00248126 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 04:59:57,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 04:59:57,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777604 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:00:01,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:00:01,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150255 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:07,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:07,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977439 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:00:11,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:00:11,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016195 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:17,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:17,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761794 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:00:21,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:00:21,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00273041 s, 1.5 MB/s\n'; <rc> = 0 >Thread-13789::WARNING::2013-03-17 05:00:25,537::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:00:25,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:00:27,681::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:00:27,683::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:00:27,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:27,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000781595 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:27,738::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000781595 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:00:27,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:00:27,738::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:00:27,740::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:00:27,741::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:27,742::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:27,742::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:27,743::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:27,743::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:27,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:28,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:28,167::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:00:28,169::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:00:28,170::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:00:28,171::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:00:28,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:28,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:28,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:28,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748332 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:00:31,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:00:31,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235486 s, 174 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:00:35,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:00:35,545::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:00:36,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:00:36,050::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:00:36,051::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:00:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:38,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159673 s, 257 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:00:41,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:00:41,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178069 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:48,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:48,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089972 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:00:51,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:00:51,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148758 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:00:58,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:00:58,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794937 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:01,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:01,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101414 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:01:08,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:01:08,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832644 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:11,764::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:01:11,783::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:01:11,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:11,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00186686 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:11,815::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00186686 s, 1.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:01:11,815::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:01:11,816::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:01:11,818::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:01:11,819::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:01:11,820::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:01:11,821::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:01:11,821::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:01:11,822::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:01:11,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:12,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:12,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:01:12,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:01:12,259::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:01:12,260::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:01:12,260::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:01:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:12,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:12,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:12,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233622 s, 175 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:01:18,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:01:18,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970707 s, 422 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:22,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:22,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847824 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:01:28,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:01:29,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834959 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:32,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:32,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803048 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13790::WARNING::2013-03-17 05:01:36,087::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:01:36,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:01:39,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:01:39,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834064 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:42,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:42,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808111 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:01:46,092::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:01:46,094::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:01:46,532::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:01:46,538::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:01:46,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:01:49,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:01:49,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00530965 s, 771 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:01:52,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:01:52,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110376 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:01:59,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:01:59,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00854157 s, 480 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:02:02,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:02:02,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204195 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:02:09,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:02:09,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843001 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:02:13,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:02:13,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148618 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:02:19,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:02:19,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796473 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:02:23,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:02:23,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015528 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:02:29,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:02:29,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014583 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:02:33,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:02:33,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724648 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:02:39,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:02:39,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704613 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:02:43,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:02:43,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111565 s, 3.7 MB/s\n'; <rc> = 0 >Thread-13791::WARNING::2013-03-17 05:02:46,566::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:02:46,570::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:02:49,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:02:49,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171072 s, 239 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:02:53,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:02:53,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842031 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:02:56,576::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:02:56,578::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:02:56,579::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:02:57,003::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:02:57,010::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:02:57,012::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:02:59,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:02:59,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00976807 s, 419 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:03:03,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:03:03,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015566 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:03:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:03:09,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840981 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:03:13,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:03:13,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100432 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:03:19,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:03:19,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773349 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:03:23,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:03:23,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779322 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:03:29,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:03:29,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081571 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:03:33,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:03:33,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300832 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:03:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:03:40,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193223 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:03:43,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:03:43,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167862 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:03:50,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:03:50,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102697 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:03:53,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:03:53,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152318 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13792::WARNING::2013-03-17 05:03:57,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:03:57,045::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:04:00,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:04:00,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811968 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:04:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:04:03,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757374 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:04:07,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:04:07,053::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:04:07,445::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:04:07,452::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:04:07,453::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:04:10,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:04:10,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174267 s, 235 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:04:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:04:13,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183547 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:04:20,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:04:20,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943206 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:04:23,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:04:23,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079113 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:04:30,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:04:30,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914009 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:04:33,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:04:33,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834546 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:04:40,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:04:40,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200883 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:04:43,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:04:43,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121659 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:04:50,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:04:50,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796908 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:04:53,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:04:53,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139571 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:00,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:00,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00268803 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:05:03,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:05:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832348 s, 4.9 MB/s\n'; <rc> = 0 >Thread-13793::WARNING::2013-03-17 05:05:07,480::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:05:07,486::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:05:10,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:10,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161541 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:05:13,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:05:13,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169898 s, 241 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:05:17,489::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:05:17,491::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:05:17,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:05:17,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:05:17,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:20,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:20,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014784 s, 277 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:05:23,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:05:23,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826008 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:30,839::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:05:30,841::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:05:30,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:30,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00091905 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:30,869::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00091905 s, 2.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:05:30,869::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:05:30,870::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:05:30,871::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:05:30,872::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:30,873::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:30,874::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:30,874::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:30,874::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:30,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:31,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:31,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:05:31,299::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:05:31,301::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:05:31,303::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:05:31,303::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:05:31,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:31,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:31,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:31,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00837264 s, 489 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:05:33,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:05:33,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133218 s, 307 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:41,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:41,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161086 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:05:43,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:05:43,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858751 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:05:51,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:05:51,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161029 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:05:53,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:05:53,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764539 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:06:01,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:06:02,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113361 s, 361 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:04,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:04,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703319 s, 582 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:06:12,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:06:12,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177622 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:14,048::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:06:14,049::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:06:14,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:14,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138309 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:14,091::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138309 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:06:14,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:06:14,092::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:06:14,093::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:06:14,098::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:06:14,099::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:06:14,099::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:06:14,099::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:06:14,100::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:06:14,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:14,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:14,504::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:06:14,505::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:06:14,508::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:06:14,509::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:06:14,510::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:06:14,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:14,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:14,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:14,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172064 s, 238 kB/s\n'; <rc> = 0 >Thread-13794::WARNING::2013-03-17 05:06:17,964::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:06:17,968::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:06:22,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:06:22,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793658 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:24,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:24,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998775 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:06:27,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:06:27,971::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:06:28,388::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:06:28,395::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:06:28,396::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:06:32,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:06:32,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825312 s, 496 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:34,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:35,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145963 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:06:42,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:06:42,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153619 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:45,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220425 s, 186 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:06:52,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:06:52,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174564 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:06:55,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:06:55,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849496 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:07:02,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:07:02,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152037 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:07:05,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:07:05,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822664 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:07:12,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:07:12,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078837 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:07:15,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:07:15,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132168 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:07:22,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:07:22,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100428 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:07:25,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:07:25,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146209 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13795::WARNING::2013-03-17 05:07:28,419::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:07:28,422::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:07:32,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:07:32,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100516 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:07:35,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:07:35,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838883 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:07:38,425::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:07:38,426::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:07:38,836::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:07:38,843::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:07:38,844::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:07:42,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:07:42,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146506 s, 280 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:07:45,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:07:45,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802844 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:07:52,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:07:52,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856134 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:07:55,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:07:55,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080982 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:08:03,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:08:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901339 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:08:05,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:08:05,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107298 s, 382 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:08:13,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:08:13,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927861 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:08:15,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:08:15,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148855 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:08:23,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:08:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176368 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:08:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:08:25,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806223 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:08:33,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:08:33,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138245 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:08:35,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:08:35,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104594 s, 3.9 MB/s\n'; <rc> = 0 >Thread-13796::WARNING::2013-03-17 05:08:38,871::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:08:38,875::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:08:43,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:08:43,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103554 s, 396 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:08:45,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:08:45,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134521 s, 3.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:08:48,877::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:08:48,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:08:48,880::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:08:49,296::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:08:49,304::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:08:49,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:08:53,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:08:53,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134754 s, 304 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:08:55,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:08:55,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014256 s, 287 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:09:03,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:09:03,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770282 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:09:05,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:09:05,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821037 s, 499 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:09:13,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:09:13,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195425 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:09:15,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:09:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160665 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:09:23,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:09:23,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154797 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:09:25,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:09:25,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146519 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:09:33,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:09:33,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133187 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:09:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:09:35,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104061 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:09:43,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:09:43,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922384 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:09:45,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:09:45,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105329 s, 3.9 MB/s\n'; <rc> = 0 >Thread-13797::WARNING::2013-03-17 05:09:49,338::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:09:49,344::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:09:53,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:09:53,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135326 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:09:55,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:09:55,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016389 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:09:59,346::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:09:59,348::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:09:59,742::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:09:59,749::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:09:59,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:04,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:04,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0332613 s, 123 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:10:05,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:10:06,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00273734 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:14,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:14,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015553 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:10:16,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:10:16,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720555 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:24,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:24,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116561 s, 351 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:10:26,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:10:26,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141332 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:34,306::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:10:34,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:10:34,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:34,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144559 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:34,356::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144559 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:10:34,357::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:10:34,358::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:10:34,359::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:10:34,361::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:34,361::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:34,362::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:34,362::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:34,362::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:34,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:34,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:34,766::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:10:34,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:10:34,769::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:10:34,770::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:10:34,771::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:10:34,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:35,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:35,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:35,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276418 s, 148 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:10:36,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:10:36,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00546283 s, 750 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:45,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:45,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842533 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:10:46,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:10:46,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118291 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:10:55,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:10:55,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691474 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:10:56,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:10:56,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183719 s, 2.2 MB/s\n'; <rc> = 0 >Thread-13798::WARNING::2013-03-17 05:10:59,775::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:10:59,782::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:11:05,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:11:05,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944369 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:06,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:06,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100852 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:11:09,785::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:11:09,786::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:11:10,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:11:10,210::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:11:10,212::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:11:15,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:11:15,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0078425 s, 522 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:16,420::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:11:16,422::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:11:16,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:16,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00174504 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:16,490::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00174504 s, 1.2 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:11:16,492::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:11:16,492::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:11:16,493::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:11:16,494::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:11:16,495::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:11:16,495::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:11:16,496::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:11:16,497::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:11:16,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:16,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:16,920::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:11:16,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:11:16,925::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:11:16,926::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:11:16,927::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:11:16,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:17,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:17,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:17,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134423 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:11:25,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:11:25,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157584 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:27,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:27,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763763 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:11:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:11:35,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243177 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:37,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:37,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210085 s, 195 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:11:45,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:11:45,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938921 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:47,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:47,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150666 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:11:55,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:11:56,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721854 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:11:57,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:11:57,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173332 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:12:06,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:12:06,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122181 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:12:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:12:07,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163568 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13799::WARNING::2013-03-17 05:12:10,238::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:12:10,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:12:16,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:12:16,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796049 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:12:17,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:12:17,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891971 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:12:20,250::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:12:20,251::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:12:20,709::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:12:20,715::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:12:20,717::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:12:26,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:12:26,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139485 s, 294 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:12:27,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:12:27,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00627283 s, 653 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:12:36,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:12:36,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155854 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:12:37,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:12:37,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878892 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:12:46,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:12:46,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793396 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:12:47,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:12:47,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154125 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:12:56,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:12:56,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176276 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:12:57,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:12:57,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877039 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:13:06,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:13:06,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144708 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:13:07,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:13:07,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085292 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:13:16,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:13:16,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157904 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:13:17,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:13:17,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141406 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13800::WARNING::2013-03-17 05:13:20,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:13:20,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:13:26,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:13:26,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800432 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:13:27,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:13:28,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079054 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:13:30,756::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:13:30,757::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:13:31,247::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:13:31,253::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:13:31,255::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:13:36,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:13:36,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0075552 s, 542 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:13:38,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:13:38,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018535 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:13:46,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:13:46,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781516 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:13:48,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:13:48,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885699 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:13:56,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:13:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807233 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:13:58,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:13:58,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104483 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:14:07,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:14:07,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843043 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:14:08,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:14:08,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139879 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:14:17,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:14:17,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785448 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:14:18,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:14:18,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888776 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:14:27,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:14:27,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104796 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:14:28,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:14:28,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190108 s, 2.2 MB/s\n'; <rc> = 0 >Thread-13801::WARNING::2013-03-17 05:14:31,282::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:14:31,286::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:14:37,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:14:37,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149936 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:14:38,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:14:38,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890458 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:14:41,288::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:14:41,290::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:14:41,292::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:14:41,779::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:14:41,786::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:14:41,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:14:47,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:14:47,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189301 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:14:48,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:14:48,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00346857 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:14:57,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:14:57,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160337 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:14:58,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:14:58,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735178 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:07,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:07,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083023 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:15:08,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:08,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178779 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:17,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:17,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715048 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:15:18,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:18,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015244 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:27,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:27,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713384 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:15:28,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:28,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155252 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:37,802::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:15:37,803::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:15:37,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:37,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0211814 s, 96.7 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:37,893::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0211814 s, 96.7 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:15:37,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:15:37,894::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:15:37,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:15:37,897::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:15:37,898::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:15:37,898::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:15:37,899::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:15:37,899::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:15:37,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:38,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:15:38,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:15:38,385::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:15:38,386::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:15:38,390::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:15:38,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:38,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:38,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109783 s, 373 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:38,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:38,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:38,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825571 s, 496 kB/s\n'; <rc> = 0 >Thread-13802::WARNING::2013-03-17 05:15:41,815::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:15:41,818::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:15:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:48,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896692 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:48,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:48,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862925 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:15:51,823::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:15:51,825::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:15:52,368::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:15:52,375::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:15:52,377::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:15:58,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:15:58,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743078 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:15:58,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:15:59,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182334 s, 225 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:08,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:08,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898013 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:16:09,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:16:09,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155984 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:18,730::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:16:18,731::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:16:18,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:18,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145286 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:18,771::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145286 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:16:18,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:16:18,772::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:16:18,774::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:16:18,775::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:16:18,775::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:16:18,776::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:16:18,776::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:16:18,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:16:18,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 05:16:19,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:19,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:19,235::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:16:19,240::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:16:19,254::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-20::DEBUG::2013-03-17 05:16:19,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134307 s, 305 kB/s\n'; <rc> = 0 >Thread-22::WARNING::2013-03-17 05:16:19,257::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:16:19,260::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:16:19,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:19,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:19,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:19,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127962 s, 320 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:16:29,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:16:29,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903586 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:29,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:29,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157437 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:16:39,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:16:39,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014267 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:39,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:39,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864031 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:16:49,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:16:49,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037776 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:49,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:49,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289724 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13803::WARNING::2013-03-17 05:16:52,405::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:16:52,410::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:16:59,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:16:59,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082777 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:16:59,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:16:59,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786938 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:17:02,412::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:17:02,413::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:17:02,896::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:17:02,903::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:17:02,904::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:17:09,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:17:09,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155527 s, 263 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:17:09,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:17:10,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011452 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:17:19,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:17:19,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739742 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:17:20,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:17:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969882 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:17:29,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:17:29,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942674 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:17:30,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:17:30,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779255 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:17:39,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:17:40,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074306 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:17:40,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:17:40,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942441 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:17:50,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:17:50,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148317 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:17:50,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:17:50,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201854 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:18:00,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:18:00,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:18:00,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106503 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:18:00,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899427 s, 4.6 MB/s\n'; <rc> = 0 >Thread-13804::WARNING::2013-03-17 05:18:02,936::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:18:02,940::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:18:10,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:18:10,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104846 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:18:10,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:18:10,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769323 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:18:12,948::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:18:12,949::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:18:13,385::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:18:13,391::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:18:13,393::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:18:20,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:18:20,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786788 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:18:20,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:18:20,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849362 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:18:30,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:18:30,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139993 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:18:30,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:18:30,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154332 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:18:40,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:18:40,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131161 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:18:40,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:18:40,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145614 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:18:50,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:18:50,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803778 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:18:50,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:18:50,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157544 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:19:00,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:19:00,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775549 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:19:00,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:19:00,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144518 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:19:10,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:19:10,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998577 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:19:10,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:19:10,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175153 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13805::WARNING::2013-03-17 05:19:13,420::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:19:13,424::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:19:20,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:19:20,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170171 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:19:20,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:19:20,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100516 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:19:23,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:19:23,428::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:19:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:19:23,883::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:19:23,884::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:19:30,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:19:30,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089619 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:19:30,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:19:30,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816235 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:19:40,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:19:40,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133244 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:19:41,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:19:41,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779659 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:19:50,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:19:50,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00319839 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:19:51,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:19:51,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152391 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:20:00,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:20:00,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010034 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:01,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:01,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146151 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:20:10,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:20:10,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316297 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:11,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:11,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00434689 s, 942 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:20:20,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:20:20,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000621822 s, 6.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:21,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:21,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143931 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13806::WARNING::2013-03-17 05:20:23,926::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:20:23,931::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:20:30,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:20:30,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902454 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:31,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:31,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00601094 s, 681 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:20:33,933::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:20:33,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:20:33,935::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:20:34,337::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:20:34,343::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:20:34,345::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:20:40,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:20:40,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835259 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:41,482::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:20:41,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:20:41,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:41,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00503248 s, 407 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:41,518::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00503248 s, 407 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:20:41,519::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:20:41,519::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:20:41,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:20:41,522::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:20:41,523::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:20:41,523::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:20:41,524::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:20:41,524::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:20:41,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:41,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:41,960::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:20:41,961::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:20:41,964::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:20:41,965::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:20:41,966::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:20:41,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:42,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:42,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:42,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00883056 s, 464 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:20:50,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:20:50,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886816 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:20:52,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:20:52,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827992 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:00,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:01,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913863 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:21:02,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:21:02,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167769 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:11,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135209 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:21:12,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:21:12,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167034 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:21,076::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:21:21,077::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:21:21,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:21,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00074237 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:21,109::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00074237 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:21:21,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:21:21,109::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:21:21,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:21:21,113::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:21,114::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:21,114::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:21,115::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:21,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:21,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:21,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:21,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:21,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:21:21,538::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:21:21,540::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:21:21,540::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:21:21,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:21,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:21,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:21,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00872155 s, 470 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:21:22,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:21:22,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00561734 s, 729 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:21:31,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:31,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978909 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:21:32,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:21:32,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977559 s, 4.2 MB/s\n'; <rc> = 0 >Thread-13807::WARNING::2013-03-17 05:21:34,370::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:21:34,376::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:21:41,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:42,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888713 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:21:42,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:21:42,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912993 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:21:44,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:21:44,380::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:21:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:21:44,853::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:21:44,854::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:21:52,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:21:52,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850161 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:21:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:21:52,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859959 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:22:02,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:22:02,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172047 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:22:02,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:22:03,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847528 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:22:12,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:22:12,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859629 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:22:13,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:22:13,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145328 s, 282 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:22:22,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:22:22,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146032 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:22:23,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:22:23,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147736 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:22:32,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:22:32,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140171 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:22:33,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:22:33,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784743 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:22:42,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:22:42,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721727 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:22:43,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:22:43,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158962 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13808::WARNING::2013-03-17 05:22:44,879::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:22:44,884::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:22:52,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:22:52,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027275 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:22:53,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:22:53,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284982 s, 1.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:22:54,887::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:22:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:22:55,294::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:22:55,301::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:22:55,302::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:23:02,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:23:02,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964356 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:23:03,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:23:03,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792314 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:23:12,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:23:12,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167812 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:23:13,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:23:13,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154418 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:23:22,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:23:22,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154194 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:23:23,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:23:23,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100635 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:23:32,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:23:32,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773476 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:23:33,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:23:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142217 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:23:42,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:23:42,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757635 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:23:43,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:23:43,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752607 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:23:52,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:23:52,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737833 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:23:54,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:23:54,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150683 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13809::WARNING::2013-03-17 05:23:55,335::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:23:55,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:24:02,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:24:02,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909633 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:24:04,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:24:04,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102985 s, 398 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:24:05,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:24:05,342::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:24:05,811::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:24:05,817::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:24:05,819::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:24:12,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:24:12,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162459 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:24:14,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:24:14,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982205 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:24:22,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:24:22,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152086 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:24:24,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:24:24,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935679 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:24:32,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:24:32,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088573 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:24:34,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:24:34,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109091 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:24:42,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:24:42,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021876 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:24:44,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:24:44,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116957 s, 350 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:24:52,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:24:52,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167045 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:24:54,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:24:54,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987593 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:25:02,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:25:02,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379437 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:04,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:04,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00742423 s, 552 kB/s\n'; <rc> = 0 >Thread-13810::WARNING::2013-03-17 05:25:05,850::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:25:05,855::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:25:12,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:25:13,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849009 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:14,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:14,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837032 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:25:15,857::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:25:15,858::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:25:16,315::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:25:16,322::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:25:16,324::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:25:23,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:25:23,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765751 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:24,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:24,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242643 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:25:33,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:25:33,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819064 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:34,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:35,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149271 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:25:43,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:25:43,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021663 s, 189 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:45,073::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:25:45,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:25:45,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:45,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0207304 s, 98.8 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:45,124::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0207304 s, 98.8 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:25:45,124::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:25:45,124::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:25:45,126::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:25:45,127::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:25:45,128::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:25:45,128::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:25:45,129::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:25:45,129::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:25:45,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:45,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:45,622::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:25:45,623::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:25:45,625::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:25:45,626::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:25:45,627::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:25:45,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:45,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:46,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:46,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134098 s, 305 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:25:53,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:25:53,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160724 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:25:56,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:25:56,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150806 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:03,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:03,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206164 s, 199 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:26:06,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:06,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150165 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:13,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:13,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00440204 s, 930 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:26:16,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:16,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810127 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13811::WARNING::2013-03-17 05:26:16,350::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:26:16,359::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 05:26:23,334::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:26:23,335::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:26:23,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:23,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00147614 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:23,369::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00147614 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:26:23,370::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:26:23,370::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:26:23,372::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:26:23,373::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:23,374::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:23,374::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:23,374::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:23,375::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:23,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:23,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:23,780::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:23,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:26:23,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:26:23,784::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:26:23,784::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:26:23,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:24,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:24,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:24,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137167 s, 299 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:26:26,373::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:26:26,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:26:26,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:26,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:26,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144686 s, 283 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:26:26,885::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:26:26,891::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:26:26,893::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:26:34,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:34,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906769 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:26:36,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:36,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853168 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:44,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:44,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101208 s, 405 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:26:46,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:46,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228573 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:26:54,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:26:54,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155568 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:26:56,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:26:56,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870537 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:27:04,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:27:04,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137565 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:27:06,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:27:06,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138418 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:27:14,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:27:14,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140265 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:27:16,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:27:16,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135074 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:27:24,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:27:24,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994258 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:27:26,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-13812::WARNING::2013-03-17 05:27:26,938::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:27:26,964::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:27:26,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185074 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:27:34,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:27:34,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804382 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:27:36,969::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:27:36,970::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 05:27:37,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:27:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00971385 s, 422 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:27:37,457::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:27:37,462::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:27:37,463::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:27:44,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:27:44,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693922 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:27:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:27:47,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843688 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:27:54,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:27:54,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783367 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:27:57,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:27:57,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146227 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:28:04,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:28:04,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171419 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:28:07,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:07,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842388 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:28:14,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:28:14,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153222 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:28:17,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:17,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148335 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:28:24,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:28:24,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866784 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:28:27,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0279196 s, 147 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:28:34,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:28:34,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168432 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13813::WARNING::2013-03-17 05:28:37,491::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:28:37,494::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:28:37,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:37,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746423 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:28:44,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:28:44,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858622 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:28:47,496::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:28:47,498::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:47,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:47,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107252 s, 382 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:28:47,948::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:28:47,954::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:28:47,955::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:28:54,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:28:54,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849249 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:28:57,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:28:57,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842462 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:29:04,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:29:04,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087314 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:29:07,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:07,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014217 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:29:14,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:29:14,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139246 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:29:17,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:17,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086765 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:29:24,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:29:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818399 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:29:27,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:27,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152948 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:29:34,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:29:35,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169184 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:29:38,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:38,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145994 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:29:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:29:45,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011403 s, 3.6 MB/s\n'; <rc> = 0 >Thread-13814::WARNING::2013-03-17 05:29:48,001::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:29:48,007::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:29:48,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:48,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152772 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:29:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:29:55,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910777 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:29:58,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:29:58,012::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:58,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:29:58,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00981088 s, 417 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:29:58,462::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:29:58,469::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:29:58,471::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:30:05,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:30:05,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015184 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:08,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:08,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014282 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:30:15,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:30:15,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192579 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:18,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:18,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918074 s, 446 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:30:25,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:30:25,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010403 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:28,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:28,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972819 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:30:35,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:30:35,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843295 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:38,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:38,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163532 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:30:45,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:30:45,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117326 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:48,631::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:30:48,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:30:48,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:48,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00913737 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:48,683::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00913737 s, 224 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:30:48,683::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:30:48,684::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:30:48,685::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:30:48,686::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:30:48,687::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:30:48,687::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:30:48,687::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:30:48,688::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:30:48,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:49,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:49,092::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:30:49,093::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:30:49,096::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:30:49,097::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:30:49,098::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:30:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:49,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:30:49,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:49,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172758 s, 237 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:30:55,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:30:55,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860743 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13815::WARNING::2013-03-17 05:30:58,506::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:30:58,511::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:30:59,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:30:59,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154814 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:05,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:05,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869006 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:31:08,514::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:31:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:31:08,945::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:31:08,952::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:31:08,953::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:31:09,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:31:09,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120311 s, 340 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:15,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:15,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139216 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:31:19,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:31:19,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155767 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:25,418::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:31:25,419::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:31:25,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:25,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000717504 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:25,449::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000717504 s, 2.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:31:25,449::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:31:25,449::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:31:25,451::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:31:25,452::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:31:25,453::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:31:25,453::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:31:25,453::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:31:25,454::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:31:25,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:25,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:25,889::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:31:25,890::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:31:25,892::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:31:25,894::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:31:25,894::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:31:25,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:26,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:26,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:26,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138249 s, 296 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:31:29,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:31:29,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203331 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:36,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:36,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821402 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:31:39,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:31:39,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171369 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:46,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:46,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845673 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:31:50,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:31:50,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010817 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:31:56,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:31:56,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889564 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:32:00,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:32:00,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013779 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:32:06,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:32:06,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010196 s, 4.0 MB/s\n'; <rc> = 0 >Thread-13816::WARNING::2013-03-17 05:32:08,982::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:32:08,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:32:10,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:32:10,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786708 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:32:16,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:32:16,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077558 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:32:18,993::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:32:18,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:32:18,996::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:32:19,412::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:32:19,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:32:19,421::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:32:20,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:32:20,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00708145 s, 578 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:32:26,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:32:26,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091969 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:32:30,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:32:30,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814628 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:32:36,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:32:36,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845444 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:32:40,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:32:40,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970098 s, 422 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:32:46,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:32:46,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137705 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:32:50,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:32:50,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833976 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:32:56,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:32:56,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151173 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:33:00,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:33:00,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00564263 s, 726 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:33:06,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:33:06,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735126 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:33:10,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:33:10,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901548 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:33:16,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:33:16,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140282 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13817::WARNING::2013-03-17 05:33:19,451::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:33:19,454::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:33:20,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:33:20,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00949682 s, 431 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:33:26,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:33:26,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846944 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:33:29,456::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:33:29,457::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:33:29,853::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:33:29,859::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:33:29,861::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:33:30,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:33:30,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128434 s, 319 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:33:36,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:33:36,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101491 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:33:40,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:33:40,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855046 s, 479 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:33:46,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:33:46,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741302 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:33:50,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:33:50,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131629 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:33:56,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:33:56,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132465 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:34:01,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:34:01,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165122 s, 248 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:34:06,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:34:06,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751389 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:34:11,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:34:11,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892215 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:34:16,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:34:17,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155001 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:34:21,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:34:21,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168398 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:34:27,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:34:27,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159035 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13818::WARNING::2013-03-17 05:34:29,884::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:34:29,887::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:34:31,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:34:31,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822982 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:34:37,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:34:37,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010696 s, 3.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:34:39,889::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:34:39,891::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:34:40,304::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:34:40,310::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:34:40,312::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:34:41,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:34:41,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142009 s, 288 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:34:47,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:34:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923918 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:34:51,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:34:51,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879025 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:34:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:34:57,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155016 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:01,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:01,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130253 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:35:07,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:35:07,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722988 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:11,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:11,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991467 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:35:17,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:35:17,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00220653 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:21,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:21,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152942 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:35:27,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:35:27,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769801 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:31,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:31,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729572 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:35:37,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:35:37,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131751 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13819::WARNING::2013-03-17 05:35:40,343::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:35:40,352::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:35:41,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:41,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157259 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:35:47,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:35:47,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726403 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:35:50,356::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:35:50,357::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:35:50,773::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:35:50,780::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:35:50,781::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:51,917::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:35:51,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:35:51,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:51,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0203322 s, 101 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:51,972::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0203322 s, 101 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:35:51,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:35:51,973::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:35:51,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:35:51,977::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:51,977::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:51,978::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:51,978::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:51,979::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:51,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:52,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:52,336::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:35:52,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:35:52,340::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:35:52,341::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:35:52,341::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:35:52,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:52,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:35:52,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:35:52,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119296 s, 343 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:35:57,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:35:57,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145764 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:36:02,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:36:02,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881367 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:07,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217093 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:36:12,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:36:12,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816067 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:17,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:17,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126398 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:36:22,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:36:22,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144396 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:27,670::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:36:27,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:36:27,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:27,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00300268 s, 682 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:27,705::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00300268 s, 682 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:36:27,706::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:36:27,706::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:36:27,708::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:36:27,709::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:36:27,709::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:36:27,710::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:36:27,710::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:36:27,711::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:36:27,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:28,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:28,154::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:36:28,155::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:36:28,157::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:36:28,159::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:36:28,159::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:36:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:28,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:28,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:28,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749319 s, 547 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:36:32,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:36:33,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108732 s, 377 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:38,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:38,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143209 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:36:43,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:36:43,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769196 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:48,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:48,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131871 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13820::WARNING::2013-03-17 05:36:50,805::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:36:50,809::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:36:53,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:36:53,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950199 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:36:58,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:36:58,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836417 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:37:00,811::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:37:00,812::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:37:01,240::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:37:01,247::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:37:01,248::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:37:03,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:37:03,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103025 s, 398 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:37:08,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:37:08,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813086 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:37:13,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:37:13,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141822 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:37:18,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:37:18,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872091 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:37:23,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:37:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865523 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:37:28,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:37:28,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799556 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:37:33,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:37:33,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224665 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:37:38,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:37:38,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00584575 s, 701 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:37:43,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:37:43,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227427 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:37:48,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:37:48,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151877 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:37:53,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:37:53,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824387 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:37:58,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:37:58,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126074 s, 3.2 MB/s\n'; <rc> = 0 >Thread-13821::WARNING::2013-03-17 05:38:01,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:38:01,286::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:38:03,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:38:03,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833216 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:38:08,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:38:09,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720503 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:38:11,294::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:38:11,295::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:38:11,296::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:38:11,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:38:11,739::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:38:11,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:38:13,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:38:13,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015698 s, 261 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:38:19,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:38:19,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898215 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:38:23,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:38:23,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758172 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:38:29,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:38:29,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115655 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:38:33,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:38:33,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135881 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:38:39,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:38:39,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154095 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:38:44,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:38:44,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124527 s, 329 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:38:49,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:38:49,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864079 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:38:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:38:54,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163733 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:38:59,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:38:59,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832383 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:39:04,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:39:04,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153308 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:39:09,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:39:09,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145564 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13822::WARNING::2013-03-17 05:39:11,767::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:39:11,772::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:39:14,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:39:14,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119703 s, 342 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:39:19,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:39:19,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768674 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:39:21,775::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:39:21,776::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:39:22,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:39:22,208::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:39:22,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:39:24,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:39:24,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251724 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:39:29,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:39:29,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834027 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:39:34,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:39:34,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851406 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:39:39,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:39:39,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372052 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:39:44,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:39:44,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705881 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:39:49,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:39:49,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161537 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:39:54,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:39:54,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842446 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:39:59,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:39:59,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887216 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:04,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:04,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882713 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:40:09,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:40:09,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874032 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:14,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:14,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001539 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:40:19,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:40:19,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173316 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13823::WARNING::2013-03-17 05:40:22,234::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:40:22,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:40:24,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:24,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848597 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:40:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:40:29,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137816 s, 3.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:40:32,245::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:40:32,247::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:40:32,666::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:40:32,673::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:40:32,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:34,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:34,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017881 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:40:39,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:40:39,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163334 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:44,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:45,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787747 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:40:49,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:40:49,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148978 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:55,053::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:40:55,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:40:55,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:55,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0148886 s, 138 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:55,109::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0148886 s, 138 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:40:55,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:40:55,110::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:40:55,112::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:40:55,113::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:55,114::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:55,114::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:55,114::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:55,116::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:55,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:55,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:55,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:40:55,576::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:40:55,579::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:40:55,580::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:40:55,580::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:40:55,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:40:55,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:40:56,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111859 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:40:59,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:40:59,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142198 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:41:06,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:41:06,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835224 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:09,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:09,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145775 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:41:16,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:41:16,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123369 s, 332 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:19,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:19,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135371 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:41:26,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:41:26,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144101 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:29,870::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:41:29,872::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:41:29,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:29,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000799859 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:29,926::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000799859 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:41:29,927::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:41:29,927::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:41:29,931::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:41:29,932::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:41:29,933::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:41:29,934::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:41:29,935::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:41:29,935::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:41:29,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:30,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:30,366::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:41:30,367::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:41:30,370::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:41:30,373::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:41:30,374::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:41:30,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:30,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:30,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:30,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0060171 s, 681 kB/s\n'; <rc> = 0 >Thread-13824::WARNING::2013-03-17 05:41:32,713::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:41:32,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:41:36,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:41:36,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153357 s, 267 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:40,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:40,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136939 s, 3.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:41:42,729::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:41:42,730::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:41:43,177::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:41:43,184::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:41:43,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:41:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:41:46,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130379 s, 314 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:41:50,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:41:50,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923951 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:41:56,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:41:56,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00903078 s, 454 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:42:00,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:42:00,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895293 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:42:06,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:42:06,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987127 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:42:10,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:42:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871894 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:42:16,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:42:16,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172385 s, 238 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:42:21,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:42:21,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144457 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:42:26,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:42:26,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137196 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:42:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:42:31,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092836 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:42:36,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:42:36,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014958 s, 274 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:42:41,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:42:41,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091422 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13825::WARNING::2013-03-17 05:42:43,213::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:42:43,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:42:46,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:42:46,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751361 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:42:51,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:42:51,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895042 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:42:53,220::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:42:53,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:42:53,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:42:53,644::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:42:53,646::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:42:56,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:42:57,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0086954 s, 471 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:43:01,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:43:01,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854302 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:43:07,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:43:07,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153711 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:43:11,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:43:11,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853719 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:43:17,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:43:17,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00903552 s, 453 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:43:21,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:43:21,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859443 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:43:27,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:43:27,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869457 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:43:31,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:43:31,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911801 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:43:37,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:43:37,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00761989 s, 538 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:43:41,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:43:41,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159498 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:43:47,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:43:47,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175059 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:43:51,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:43:51,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132444 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13826::WARNING::2013-03-17 05:43:53,687::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:43:53,690::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:43:57,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:43:57,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00806396 s, 508 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:44:01,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:44:01,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766042 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:44:03,692::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:44:03,693::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:44:03,694::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:44:04,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:44:04,153::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:44:04,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:44:07,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:44:07,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110826 s, 370 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:44:11,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:44:11,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914846 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:44:17,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:44:17,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00873678 s, 469 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:44:21,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:44:21,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230662 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:44:27,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:44:27,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826349 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:44:31,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:44:31,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840414 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:44:37,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:44:37,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00692549 s, 591 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:44:41,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:44:41,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139766 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:44:47,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:44:47,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142336 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:44:51,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:44:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258957 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:44:57,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:44:58,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139278 s, 294 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:45:01,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:45:01,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123273 s, 3.3 MB/s\n'; <rc> = 0 >Thread-13827::WARNING::2013-03-17 05:45:04,182::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:45:04,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:45:08,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:08,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796364 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:45:11,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:45:11,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910132 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:45:14,192::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:45:14,193::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:45:14,669::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:45:14,676::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:45:14,678::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:18,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:18,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01101 s, 372 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:45:21,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:45:21,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140669 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:28,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:28,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989705 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:45:31,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:45:31,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865472 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:38,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:38,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741299 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:45:41,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:45:41,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766816 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:48,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:48,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703488 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:45:51,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:45:51,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799238 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:58,409::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:45:58,410::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:45:58,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:58,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0150657 s, 136 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:58,471::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0150657 s, 136 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:45:58,471::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:45:58,472::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:45:58,473::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:45:58,474::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:58,475::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:58,475::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:58,476::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:58,476::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:58,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:58,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:58,919::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:45:58,920::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:45:58,923::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:45:58,924::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:45:58,924::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:45:58,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:59,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:45:59,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:45:59,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00663423 s, 617 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:01,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:01,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161952 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:46:09,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:46:09,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799275 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:12,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:12,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122657 s, 3.3 MB/s\n'; <rc> = 0 >Thread-13828::WARNING::2013-03-17 05:46:14,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:46:14,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:46:19,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:46:19,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00331847 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:22,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:22,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792283 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:46:24,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:46:24,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:46:25,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:46:25,154::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:46:25,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:46:29,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:46:29,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169288 s, 242 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:32,079::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:46:32,080::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:46:32,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:32,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000750913 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:32,109::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000750913 s, 2.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:46:32,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:46:32,109::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:46:32,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:46:32,112::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:46:32,113::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:46:32,113::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:46:32,113::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:46:32,114::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:46:32,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:32,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:32,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:46:32,543::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:46:32,546::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:46:32,547::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:46:32,547::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:46:32,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:32,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:32,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:32,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180838 s, 227 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:46:39,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:46:39,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00936677 s, 437 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:42,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:43,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157131 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:46:49,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:46:49,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779932 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:46:53,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:46:53,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160781 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:46:59,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:46:59,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858385 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:47:03,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:47:03,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687615 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:47:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:47:09,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151826 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:47:13,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:47:13,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150526 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:47:20,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:47:20,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174725 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:47:23,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:47:23,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014774 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13829::WARNING::2013-03-17 05:47:25,185::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:47:25,188::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:47:30,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:47:30,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835138 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:47:33,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:47:33,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750294 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:47:35,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:47:35,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:47:35,630::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:47:35,638::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:47:35,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:47:40,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:47:40,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114844 s, 357 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:47:43,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:47:43,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120233 s, 341 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:47:50,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:47:50,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865925 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:47:53,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:47:53,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148195 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:48:00,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:48:00,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151167 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:48:03,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:48:03,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761154 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:48:10,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:48:10,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161592 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:48:13,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:48:13,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136232 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:48:20,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:48:20,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166787 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:48:23,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:48:23,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132931 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:48:30,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:48:30,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013984 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:48:33,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:48:33,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802795 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13830::WARNING::2013-03-17 05:48:35,667::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:48:35,671::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:48:40,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:48:40,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167695 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:48:43,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:48:43,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895658 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:48:45,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:48:45,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:48:46,159::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:48:46,166::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:48:46,168::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:48:50,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:48:50,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185922 s, 220 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:48:53,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:48:53,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894752 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:49:00,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:49:00,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186932 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:49:03,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:49:03,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146144 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:49:10,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:49:11,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102309 s, 400 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:49:13,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:49:13,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872067 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:49:21,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:49:21,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149251 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:49:23,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:49:23,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900045 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:49:31,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:49:31,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197285 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:49:33,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:49:33,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00834483 s, 491 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:49:41,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:49:41,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110166 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:49:43,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:49:43,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119768 s, 3.4 MB/s\n'; <rc> = 0 >Thread-13831::WARNING::2013-03-17 05:49:46,191::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:49:46,194::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:49:51,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:49:51,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0474857 s, 86.3 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:49:53,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:49:53,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810502 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:49:56,196::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:49:56,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:49:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:49:56,624::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:49:56,631::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:49:56,633::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:50:01,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:50:01,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00916009 s, 447 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:50:03,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:50:03,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854987 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:50:11,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:50:11,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156686 s, 261 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:50:13,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:50:13,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016313 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:50:21,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:50:21,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885759 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:50:23,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:50:23,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000690796 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:50:31,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:50:31,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161033 s, 254 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:50:33,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:50:34,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957583 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:50:41,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:50:41,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146612 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:50:44,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:50:44,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145491 s, 282 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:50:51,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:50:51,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168532 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:50:54,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:50:54,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138477 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13832::WARNING::2013-03-17 05:50:56,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:50:56,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:51:02,010::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:51:02,012::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:51:02,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:02,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000863086 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:02,044::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000863086 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:51:02,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:51:02,045::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:51:02,047::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:51:02,048::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:02,049::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:02,050::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:02,050::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:02,051::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:02,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:02,446::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:02,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:51:02,450::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:51:02,451::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:51:02,451::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:51:02,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:02,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:02,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:02,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165888 s, 247 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:04,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:04,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104668 s, 391 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:51:06,680::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:51:06,681::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:51:07,113::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:51:07,120::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:51:07,121::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:51:12,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:12,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151079 s, 271 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:14,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:14,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263206 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:22,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:23,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010212 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:24,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:24,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122597 s, 334 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:33,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:33,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151621 s, 270 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:34,263::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:51:34,265::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:51:34,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:34,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00156512 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:34,308::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00156512 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:51:34,309::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:51:34,309::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:51:34,311::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:51:34,313::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:51:34,313::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:51:34,314::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:51:34,314::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:51:34,314::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:51:34,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:34,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:34,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:51:34,751::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:51:34,753::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:51:34,754::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:51:34,755::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:51:34,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:35,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:35,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:35,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176155 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:43,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:43,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013226 s, 310 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:45,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:45,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132148 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:51:53,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:51:53,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123881 s, 331 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:51:55,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:51:55,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738797 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:52:03,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:52:03,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192664 s, 213 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:52:05,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:52:05,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145656 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13833::WARNING::2013-03-17 05:52:07,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:52:07,157::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:52:13,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:52:13,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116049 s, 353 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:52:15,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:52:15,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831214 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:52:17,161::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:52:17,162::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:52:17,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:52:17,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:52:17,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:52:23,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:52:23,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174657 s, 235 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:52:25,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:52:25,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137199 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:52:33,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:52:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186647 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:52:35,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:52:35,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860719 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:52:43,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:52:43,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151617 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:52:45,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:52:45,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021771 s, 188 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:52:53,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:52:53,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144685 s, 283 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:52:55,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:52:55,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00275179 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:53:03,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:53:03,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229699 s, 178 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:53:05,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:53:05,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173136 s, 237 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:53:13,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:53:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013547 s, 302 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:53:15,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:53:15,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00400013 s, 1.0 MB/s\n'; <rc> = 0 >Thread-13834::WARNING::2013-03-17 05:53:17,620::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:53:17,624::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:53:24,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:53:24,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110454 s, 371 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:53:25,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:53:25,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802271 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:53:27,626::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:53:27,628::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:53:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:53:28,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:53:28,069::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:53:34,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:53:34,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209654 s, 195 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:53:35,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:53:35,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987005 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:53:44,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:53:44,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124991 s, 328 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:53:45,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:53:45,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715573 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:53:54,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:53:54,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011946 s, 343 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:53:55,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:53:55,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820301 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:54:04,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:54:04,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013822 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:54:05,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:54:06,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176412 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:54:14,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:54:14,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150085 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:54:16,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:54:16,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139106 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:54:24,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:54:24,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001664 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:54:26,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:54:26,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130083 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13835::WARNING::2013-03-17 05:54:28,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:54:28,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:54:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:54:34,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131155 s, 312 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:54:36,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:54:36,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855351 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:54:38,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:54:38,103::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:54:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:54:38,549::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:54:38,550::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:54:44,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:54:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148526 s, 276 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:54:46,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:54:46,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929793 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:54:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:54:54,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183953 s, 223 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:54:56,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:54:56,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765034 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:55:04,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:55:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0437748 s, 93.6 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:55:06,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:55:06,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811259 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:55:15,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:55:15,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100771 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:55:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:55:16,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768289 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:55:25,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:55:25,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741639 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:55:26,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:55:26,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144972 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:55:35,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:55:35,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793139 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:55:36,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:55:36,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906473 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13836::WARNING::2013-03-17 05:55:38,581::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:55:38,586::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:55:45,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:55:45,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080978 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:55:46,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:55:46,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847545 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:55:48,592::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 05:55:48,593::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:55:48,595::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:55:49,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:55:49,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:55:49,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:55:55,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:55:55,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154745 s, 265 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:55:56,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:55:56,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925881 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:05,642::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 05:56:05,643::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:56:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:05,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00115446 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:05,697::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00115446 s, 1.8 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 05:56:05,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 05:56:05,698::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 05:56:05,699::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 05:56:05,700::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:56:05,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:56:05,701::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:56:05,702::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:56:05,702::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 05:56:05,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:06,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:06,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:56:06,143::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 05:56:06,145::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 05:56:06,146::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 05:56:06,147::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 05:56:06,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:06,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:06,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232332 s, 176 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:06,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:06,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:06,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153593 s, 267 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:16,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:16,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154806 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:16,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:16,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150352 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:26,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:26,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788577 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:26,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:26,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147726 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:36,661::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 05:56:36,662::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:56:36,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:36,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135835 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:36,703::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135835 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 05:56:36,704::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 05:56:36,704::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 05:56:36,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 05:56:36,710::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:56:36,711::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:56:36,711::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:56:36,712::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:56:36,712::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 05:56:36,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:36,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:36,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148596 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:37,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:37,149::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 05:56:37,150::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 05:56:37,152::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 05:56:37,153::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 05:56:37,154::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 05:56:37,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:37,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:37,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:37,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00454029 s, 902 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:56:46,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:47,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00846034 s, 484 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:47,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:47,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808488 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13837::WARNING::2013-03-17 05:56:49,110::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:56:49,113::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:56:57,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:56:57,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121804 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:56:57,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:56:57,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101209 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:56:59,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:56:59,116::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:56:59,567::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:56:59,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:56:59,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:57:07,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:57:07,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153714 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:57:07,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:57:07,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113256 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:57:17,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:57:17,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913968 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:57:17,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:57:17,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916158 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:57:27,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:57:27,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891896 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:57:27,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:57:27,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855748 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:57:37,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:57:37,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140511 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:57:37,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:57:37,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155747 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:57:47,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:57:47,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845003 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:57:47,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:57:47,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122293 s, 335 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:57:57,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:57:57,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131816 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:57:58,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:57:58,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120129 s, 3.4 MB/s\n'; <rc> = 0 >Thread-13838::WARNING::2013-03-17 05:57:59,603::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:57:59,607::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:58:07,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:58:07,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983022 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:58:08,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:58:08,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151824 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:58:09,610::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:58:09,612::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:58:10,048::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:58:10,054::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:58:10,056::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:58:17,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:58:17,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185547 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:58:18,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:58:18,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850922 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:58:27,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:58:27,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767713 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:58:28,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:58:28,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897372 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:58:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:58:37,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139383 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:58:38,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:58:38,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182955 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:58:47,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:58:48,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139877 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:58:48,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:58:48,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014314 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:58:58,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:58:58,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015267 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:58:58,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:58:58,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759739 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:59:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:59:08,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197665 s, 207 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:59:08,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:59:08,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146977 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13839::WARNING::2013-03-17 05:59:10,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 05:59:10,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 05:59:18,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:59:18,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790443 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:59:18,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:59:18,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870417 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 05:59:20,095::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 05:59:20,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 05:59:20,577::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 05:59:20,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 05:59:20,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 05:59:28,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:59:28,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183406 s, 223 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:59:28,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:59:28,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016008 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:59:38,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:59:38,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209211 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:59:38,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:59:38,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946793 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:59:48,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:59:48,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112247 s, 365 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:59:48,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:59:48,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090615 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 05:59:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 05:59:58,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148828 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 05:59:58,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 05:59:58,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162934 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:00:08,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:00:08,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:00:08,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876014 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:00:08,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108371 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:00:18,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:00:18,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743671 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:00:18,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:00:18,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969324 s, 4.2 MB/s\n'; <rc> = 0 >Thread-13840::WARNING::2013-03-17 06:00:20,614::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:00:20,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:00:28,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:00:28,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896872 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:00:28,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:00:28,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863973 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:00:30,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:00:30,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:00:31,041::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:00:31,048::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:00:31,050::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:00:38,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:00:38,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014912 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:00:39,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:00:39,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015373 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:00:48,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:00:49,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028841 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:00:49,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:00:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186443 s, 220 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:00:59,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:00:59,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079563 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:00:59,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:00:59,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187188 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:01:09,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:09,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176777 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:09,298::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:01:09,300::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:01:09,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.01128 s, 182 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:09,344::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.01128 s, 182 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:01:09,344::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:01:09,344::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:01:09,350::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:01:09,351::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:01:09,352::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:01:09,353::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:01:09,353::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:01:09,353::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:01:09,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:09,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:09,781::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:01:09,783::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:01:09,785::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:01:09,786::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:01:09,787::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:01:09,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:10,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:10,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:10,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00351605 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:01:19,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:19,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817897 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:20,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:20,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792926 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:01:29,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:29,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00318323 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:30,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:30,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133025 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13841::WARNING::2013-03-17 06:01:31,076::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:01:31,081::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:01:39,170::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:01:39,171::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:01:39,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:39,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000893072 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:01:39,200::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000893072 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:01:39,200::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:01:39,200::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:01:39,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:01:39,203::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:39,204::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:39,204::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:39,204::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:39,205::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:39,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:39,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:01:39,639::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:39,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:01:39,642::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:01:39,644::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:01:39,644::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:01:39,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:40,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:01:40,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:40,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274731 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:40,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:40,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00754421 s, 543 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:01:41,082::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:01:41,083::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:01:41,084::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:01:41,514::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:01:41,521::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:01:41,523::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:01:50,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:01:50,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803062 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:01:50,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:01:50,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883208 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:02:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:02:00,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769775 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:02:00,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:02:01,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841227 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:02:10,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:02:10,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153069 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:02:11,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:02:11,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856288 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:02:20,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:02:20,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204588 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:02:21,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:02:21,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769744 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:02:30,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:02:30,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794783 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:02:31,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:02:31,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143496 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:02:40,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:02:40,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110849 s, 370 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:02:41,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:02:41,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123325 s, 3.3 MB/s\n'; <rc> = 0 >Thread-13842::WARNING::2013-03-17 06:02:41,550::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:02:41,560::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:02:50,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:02:50,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800198 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:02:51,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 06:02:51,755::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:02:51,828::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:02:51,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191062 s, 214 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:02:52,346::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:02:52,351::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:02:52,355::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:03:00,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:03:00,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174148 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:03:01,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:03:01,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783447 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:03:10,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:03:10,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872306 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:03:12,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:03:12,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00481727 s, 850 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:03:20,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:03:20,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872034 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:03:22,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:03:22,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207569 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:03:30,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:03:30,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147129 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:03:32,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:03:32,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149072 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:03:40,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:03:40,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820727 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:03:42,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:03:42,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171108 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:03:50,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:03:51,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825348 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:03:52,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-13843::WARNING::2013-03-17 06:03:52,397::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-20::DEBUG::2013-03-17 06:03:52,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787753 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::ERROR::2013-03-17 06:03:52,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:04:01,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:04:01,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100519 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:04:02,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:04:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:02,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:02,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100354 s, 408 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:04:02,871::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:04:02,877::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:04:02,878::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:04:11,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:04:11,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832072 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:04:12,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:12,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156888 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:04:21,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:04:21,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880421 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:04:22,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:22,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803813 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:04:31,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:04:31,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895252 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:04:32,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:32,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105901 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:04:41,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:04:41,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243613 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:04:42,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:42,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131687 s, 311 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:04:51,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:04:51,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186952 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:04:52,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:04:52,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853719 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:05:01,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:05:01,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895993 s, 4.6 MB/s\n'; <rc> = 0 >Thread-13844::WARNING::2013-03-17 06:05:02,914::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:05:02,918::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:05:02,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:02,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091603 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:05:11,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:05:11,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917287 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:05:12,921::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:05:12,922::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:13,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:13,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159762 s, 256 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:05:13,396::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:05:13,402::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:05:13,403::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:05:21,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:05:21,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074624 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:05:23,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:23,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844555 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:05:31,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:05:31,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860016 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:05:33,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:33,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880083 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:05:41,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:05:41,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939654 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:05:43,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:43,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132859 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:05:51,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:05:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808831 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:05:53,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:05:53,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695936 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:01,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:01,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010979 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:03,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:03,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148153 s, 276 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:11,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:11,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00731177 s, 560 kB/s\n'; <rc> = 0 >Thread-13845::WARNING::2013-03-17 06:06:13,433::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:06:13,439::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:06:13,519::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:06:13,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:06:13,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:13,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014433 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:13,561::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014433 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:06:13,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:06:13,562::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:06:13,564::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:06:13,565::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:13,565::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:13,566::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:13,566::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:13,567::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:13,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:14,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:14,060::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:14,061::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:06:14,063::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:06:14,064::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:06:14,065::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:06:14,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:14,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:14,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:14,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010103 s, 405 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:21,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:21,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852541 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:06:23,442::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:06:23,443::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:06:23,864::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:06:23,870::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:06:23,872::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:06:24,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:24,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796609 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:31,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:31,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974154 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:34,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:34,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110352 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:41,898::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:06:41,900::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:06:41,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:41,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00244844 s, 836 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:41,948::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00244844 s, 836 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:06:41,948::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:06:41,948::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:06:41,950::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:06:41,951::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:06:41,951::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:06:41,952::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:06:41,952::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:06:41,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:06:41,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:42,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:42,355::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:06:42,356::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:06:42,359::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:06:42,360::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:06:42,360::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:06:42,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:42,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:42,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:42,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013761 s, 298 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:44,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:44,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736679 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:06:52,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:06:52,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134563 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:06:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:06:54,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137568 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:07:02,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:07:02,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203058 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:07:04,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:07:04,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151875 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:07:12,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:07:12,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112839 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:07:14,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:07:14,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147602 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:07:22,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:07:23,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079253 s, 5.2 MB/s\n'; <rc> = 0 >Thread-13846::WARNING::2013-03-17 06:07:23,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:07:23,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:07:25,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:07:25,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843979 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:07:33,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:07:33,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0045451 s, 901 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:07:33,903::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:07:33,904::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:07:33,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:07:34,332::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:07:34,339::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:07:34,341::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:07:35,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:07:35,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834151 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:07:43,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:07:43,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851195 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:07:45,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:07:45,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784855 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:07:53,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:07:53,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131043 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:07:55,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:07:55,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158277 s, 259 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:08:03,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:08:03,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991789 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:08:05,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:08:05,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147785 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:08:13,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:08:13,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117111 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:08:15,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:08:15,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743424 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:08:23,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:08:23,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906726 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:08:25,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:08:25,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970606 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:08:33,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:08:33,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114622 s, 3.6 MB/s\n'; <rc> = 0 >Thread-13847::WARNING::2013-03-17 06:08:34,369::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:08:34,375::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:08:35,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:08:35,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870003 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:08:43,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:08:43,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014439 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:08:44,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:08:44,381::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:08:44,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:08:44,837::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:08:44,838::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:08:45,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:08:45,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00939853 s, 436 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:08:53,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:08:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149427 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:08:55,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:08:55,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861223 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:09:03,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:09:03,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842499 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:09:05,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:09:05,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934078 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:09:13,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:09:13,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169765 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:09:15,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:09:15,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820331 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:09:23,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:09:23,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174449 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:09:26,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:09:26,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945291 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:09:33,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:09:33,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101978 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:09:36,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:09:36,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879162 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:09:44,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:09:44,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00936834 s, 437 kB/s\n'; <rc> = 0 >Thread-13848::WARNING::2013-03-17 06:09:44,861::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:09:44,867::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:09:46,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:09:46,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0048833 s, 839 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:09:54,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:09:54,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831761 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:09:54,874::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:09:54,875::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:09:55,341::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:09:55,348::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:09:55,350::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:09:56,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:09:56,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160726 s, 255 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:10:04,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:10:04,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106712 s, 384 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:10:06,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:10:06,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175544 s, 233 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:10:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:10:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952446 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:10:16,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:10:16,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830111 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:10:24,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:10:24,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119811 s, 342 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:10:26,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:10:26,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150063 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:10:34,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:10:34,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150381 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:10:36,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:10:36,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852212 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:10:44,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:10:44,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123721 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:10:46,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:10:46,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790743 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:10:54,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:10:54,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757685 s, 5.4 MB/s\n'; <rc> = 0 >Thread-13849::WARNING::2013-03-17 06:10:55,376::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:10:55,380::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:10:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:10:56,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890737 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:04,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:04,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766226 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:11:05,382::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:11:05,384::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:11:05,804::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:11:05,810::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:11:05,811::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:06,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:06,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152324 s, 269 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:14,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:14,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847011 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:16,888::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:11:16,889::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:11:16,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:16,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00089285 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:16,919::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00089285 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:11:16,920::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:11:16,920::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:11:16,922::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:11:16,923::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:16,924::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:16,924::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:16,925::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:16,925::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:16,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:17,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:17,369::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:11:17,370::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:11:17,373::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:11:17,375::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:11:17,376::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:11:17,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:17,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:17,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:17,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157659 s, 260 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:24,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:24,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809441 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:27,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:27,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822615 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:34,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:34,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146746 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:37,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:37,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103021 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:44,924::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:11:44,925::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:11:44,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:44,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00304695 s, 672 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:44,973::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00304695 s, 672 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:11:44,974::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:11:44,974::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:11:44,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:11:44,976::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:11:44,977::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:11:44,977::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:11:44,978::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:11:44,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:11:44,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:45,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:45,401::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:11:45,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:11:45,404::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:11:45,405::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:11:45,406::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:11:45,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:45,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:45,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111938 s, 366 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:47,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:48,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732178 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:11:55,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:11:55,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137436 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:11:58,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:11:58,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147232 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13850::WARNING::2013-03-17 06:12:05,840::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:12:05,843::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:12:05,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:05,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755745 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:12:08,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:12:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868292 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:12:15,845::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:12:15,846::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:15,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:15,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133961 s, 306 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:12:16,291::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:12:16,297::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:12:16,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:12:18,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:12:18,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798241 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:12:25,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:25,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913752 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:12:28,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:12:28,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00413129 s, 991 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:12:36,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147491 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:12:38,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:12:38,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976587 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:12:46,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159901 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:12:48,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:12:48,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838589 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:12:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:12:56,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165829 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:12:58,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:12:58,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136746 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:13:06,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:13:06,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106763 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:13:08,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:13:08,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017238 s, 238 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:13:16,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:13:16,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00499444 s, 820 kB/s\n'; <rc> = 0 >Thread-13851::WARNING::2013-03-17 06:13:16,345::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:13:16,349::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:13:18,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:13:18,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917288 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:13:26,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:13:26,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132147 s, 310 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:13:26,351::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:13:26,353::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:13:26,354::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:13:26,824::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:13:26,831::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:13:26,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:13:28,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:13:28,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163227 s, 251 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:13:36,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:13:36,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763834 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:13:38,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:13:38,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706576 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:13:46,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:13:46,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846549 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:13:48,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:13:48,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969321 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:13:56,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:13:56,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790358 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:13:59,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:13:59,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144731 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:14:06,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:14:06,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295743 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:14:09,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:14:09,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138388 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:14:16,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:14:16,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833395 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:14:19,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:14:19,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718026 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:14:26,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:14:26,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000676656 s, 6.1 MB/s\n'; <rc> = 0 >Thread-13852::WARNING::2013-03-17 06:14:26,859::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:14:26,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:14:29,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:14:29,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145559 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:14:36,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:14:36,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147288 s, 278 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:14:36,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:14:36,868::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:14:37,318::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:14:37,324::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:14:37,326::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:14:39,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:14:39,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115051 s, 356 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:14:46,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:14:46,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173219 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:14:49,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:14:49,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155401 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:14:56,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:14:56,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135249 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:14:59,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:14:59,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709109 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:15:06,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:15:06,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108344 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:15:09,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:15:09,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171532 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:15:16,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:15:16,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136657 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:15:19,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:15:19,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859933 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:15:26,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:15:26,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139894 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:15:29,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:15:29,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114687 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:15:36,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:15:36,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760712 s, 5.4 MB/s\n'; <rc> = 0 >Thread-13853::WARNING::2013-03-17 06:15:37,361::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:15:37,365::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:15:39,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:15:39,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741607 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:15:46,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:15:47,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0394895 s, 104 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:15:47,368::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:15:47,369::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:15:47,804::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:15:47,810::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:15:47,812::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:15:49,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:15:49,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197628 s, 207 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:15:57,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:15:57,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103335 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:15:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:00,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996055 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:07,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:07,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800092 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:10,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:10,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123592 s, 331 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:17,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:17,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132983 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:20,127::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:16:20,129::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:16:20,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:20,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138738 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:20,170::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138738 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:16:20,171::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:16:20,171::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:16:20,173::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:16:20,174::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:16:20,174::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:16:20,175::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:16:20,175::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:16:20,176::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:16:20,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:20,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:20,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:16:20,586::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:16:20,591::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:16:20,593::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:16:20,593::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:16:20,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:20,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:20,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:20,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00699385 s, 586 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:27,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:27,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193743 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:31,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137992 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:37,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:37,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013217 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:41,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:41,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106978 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:47,314::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:16:47,318::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:16:47,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:47,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144985 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:47,379::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144985 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:16:47,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:16:47,383::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:16:47,388::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:16:47,394::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:47,398::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:47,400::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:47,402::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:47,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:47,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-13854::WARNING::2013-03-17 06:16:47,849::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:16:47,855::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:16:47,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:47,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:47,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:16:47,878::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:16:47,879::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:16:47,880::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:16:47,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:48,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:16:48,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:48,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163392 s, 251 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:16:51,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:16:51,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867439 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:16:57,858::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:16:57,859::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:16:58,264::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:16:58,270::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:16:58,272::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:16:58,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:16:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117633 s, 348 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:17:01,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:17:01,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177855 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:17:08,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:17:08,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014618 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:17:11,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:17:11,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811538 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:17:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:17:18,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891619 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:17:21,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:17:21,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829608 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:17:28,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:17:28,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149493 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:17:31,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:17:31,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081651 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:17:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:17:38,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790061 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:17:41,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:17:41,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154974 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:17:48,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:17:48,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803654 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:17:51,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:17:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0074334 s, 551 kB/s\n'; <rc> = 0 >Thread-13855::WARNING::2013-03-17 06:17:58,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:17:58,329::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:17:58,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:17:58,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872463 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:18:01,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:18:01,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200485 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:18:08,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:18:08,333::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:08,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:08,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146781 s, 279 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:18:08,834::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:18:08,839::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:18:08,841::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:18:11,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:18:11,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161328 s, 254 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:18:18,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:18,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154309 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:18:22,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:18:22,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880072 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:18:28,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:28,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842553 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:18:32,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:18:32,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136657 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:18:38,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:38,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160755 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:18:42,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:18:42,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912135 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:18:48,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:48,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001654 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:18:52,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:18:52,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00770621 s, 532 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:18:58,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:18:58,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001432 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:19:02,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:19:02,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135316 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13856::WARNING::2013-03-17 06:19:08,885::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:19:08,889::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:19:08,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:08,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165493 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:19:12,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:19:12,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827351 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:19:18,891::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:19:18,892::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:19:18,893::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:18,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:19,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184004 s, 223 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:19:19,342::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:19:19,348::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:19:19,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:19:22,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:19:22,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00887528 s, 462 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:19:29,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184107 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:19:32,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:19:32,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124477 s, 329 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:19:39,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:39,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142851 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:19:42,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:19:42,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888091 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:19:49,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:49,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810538 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:19:52,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:19:52,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0094106 s, 435 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:19:59,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:19:59,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936924 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:20:02,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:20:02,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144977 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:20:09,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:20:09,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146265 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:20:12,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:20:12,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117035 s, 350 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:20:19,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:20:19,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753043 s, 5.4 MB/s\n'; <rc> = 0 >Thread-13857::WARNING::2013-03-17 06:20:19,371::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:20:19,375::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:20:23,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:20:23,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759427 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:20:29,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 06:20:29,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:20:29,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:20:29,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142453 s, 288 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:20:29,887::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:20:29,892::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:20:29,893::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:20:33,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:20:33,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120636 s, 340 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:20:39,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:20:39,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157475 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:20:43,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:20:43,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916597 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:20:49,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:20:49,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768215 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:20:53,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:20:53,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146241 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:20:59,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:20:59,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717558 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:03,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:03,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883887 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:09,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:09,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945426 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:13,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:13,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017852 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:19,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:19,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878636 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:23,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:21:23,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:21:23,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:23,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121973 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:23,501::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121973 s, 1.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:21:23,501::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:21:23,501::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:21:23,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:21:23,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:23,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:23,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:23,509::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:23,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:23,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:23,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:23,956::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:23,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:21:23,959::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:21:23,961::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:21:23,962::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:21:23,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:24,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:24,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:24,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00748639 s, 547 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:29,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:29,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00354652 s, 1.2 MB/s\n'; <rc> = 0 >Thread-13858::WARNING::2013-03-17 06:21:29,917::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:21:29,921::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:21:34,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:34,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961577 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:39,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:39,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186234 s, 220 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:21:39,924::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:21:39,925::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:21:40,349::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:21:40,356::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:21:40,357::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:21:44,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:44,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120485 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:49,783::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:21:49,784::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:21:49,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:49,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000803201 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:49,816::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000803201 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:21:49,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:21:49,816::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:21:49,818::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:21:49,819::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:21:49,820::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:21:49,820::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:21:49,820::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:21:49,821::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:21:49,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:50,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:50,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:21:50,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:21:50,214::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:21:50,215::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:21:50,216::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:21:50,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:21:50,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:21:50,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203517 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:21:54,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:21:54,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758947 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:22:00,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:22:00,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075186 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:22:04,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:22:04,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824241 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:22:10,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:22:10,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793803 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:22:14,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:22:14,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829386 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:22:20,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:22:20,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011755 s, 348 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:22:24,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:22:24,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178795 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:22:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:22:30,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916571 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:22:34,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:22:34,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149491 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13859::WARNING::2013-03-17 06:22:40,397::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:22:40,400::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:22:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:22:40,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905317 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:22:44,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:22:44,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158488 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:22:50,402::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:22:50,404::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:22:50,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 06:22:50,878::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-22::DEBUG::2013-03-17 06:22:50,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141598 s, 289 kB/s\n'; <rc> = 0 >Thread-21::WARNING::2013-03-17 06:22:50,923::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:22:50,924::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:22:55,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:22:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741047 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:23:00,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:23:00,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815538 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:23:05,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:23:05,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245021 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:23:10,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:23:10,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010605 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:23:15,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:23:15,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00257635 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:23:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:23:21,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134604 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:23:25,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:23:25,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817244 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:23:31,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:23:31,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161586 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:23:35,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:23:35,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154677 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:23:41,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:23:41,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169316 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:23:45,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:23:45,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914079 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13860::WARNING::2013-03-17 06:23:50,951::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:23:50,954::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:23:51,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:23:51,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143199 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:23:55,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:23:55,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172669 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:24:00,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:24:00,960::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:01,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:01,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01473 s, 278 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:24:01,452::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:24:01,458::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:24:01,459::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:24:05,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:24:05,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00956393 s, 428 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:24:11,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:11,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104105 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:24:15,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:24:15,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156299 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:24:21,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:21,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791338 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:24:25,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:24:25,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809666 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:24:31,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:31,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875109 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:24:35,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:24:35,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711051 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:24:41,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:41,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00705018 s, 581 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:24:45,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:24:45,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853557 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:24:51,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:24:51,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127841 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:24:56,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:24:56,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181603 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13861::WARNING::2013-03-17 06:25:01,485::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:25:01,488::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:25:01,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:01,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860592 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:25:06,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:25:06,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782577 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:25:11,492::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:25:11,493::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:25:11,495::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:11,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202573 s, 202 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:25:11,938::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:25:11,943::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:25:11,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:25:16,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:25:16,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101747 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:25:21,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:21,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070518 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:25:26,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:25:26,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757485 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:25:31,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:31,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789877 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:25:36,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:25:36,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869037 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:25:41,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:41,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00654051 s, 626 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:25:46,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:25:46,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771746 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:25:51,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:25:51,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125376 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:25:56,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:25:56,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143914 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:01,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:01,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137051 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:06,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:06,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372852 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:11,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:11,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00355964 s, 1.2 MB/s\n'; <rc> = 0 >Thread-13862::WARNING::2013-03-17 06:26:11,973::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:26:11,980::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:26:16,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:16,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104375 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:21,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:21,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139309 s, 294 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:26:21,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:26:21,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:26:22,454::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:26:22,461::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:26:22,462::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:26,891::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:26:26,894::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:26:26,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:26,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129133 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:26,959::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129133 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:26:26,960::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:26:26,960::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:26:26,962::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:26:26,963::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:26,963::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:26,964::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:26,964::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:26,965::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:26,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:27,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:27,406::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:26:27,407::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:26:27,409::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:26:27,410::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:26:27,411::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:26:27,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:27,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:27,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:27,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257762 s, 159 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:31,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:32,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019516 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:37,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:37,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708152 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:42,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135813 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:47,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:47,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143939 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:52,063::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:26:52,064::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:26:52,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:52,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000874831 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:52,096::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000874831 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:26:52,096::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:26:52,097::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:26:52,099::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:26:52,100::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:26:52,101::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:26:52,102::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:26:52,102::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:26:52,103::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:26:52,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:52,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:52,544::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:26:52,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:26:52,548::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:26:52,549::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:26:52,549::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:26:52,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:52,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:26:52,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:26:52,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200199 s, 205 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:26:58,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:26:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00686121 s, 597 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:27:02,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:27:03,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139172 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:27:08,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:27:08,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014956 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:27:13,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:27:13,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800006 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:27:18,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:27:18,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132605 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13863::WARNING::2013-03-17 06:27:22,492::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:27:22,496::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:27:23,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:27:23,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835706 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:27:28,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:27:28,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142691 s, 2.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:27:32,498::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:27:32,499::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:27:32,921::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:27:32,928::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:27:32,929::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:27:33,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:27:33,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168486 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:27:38,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:27:38,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233555 s, 175 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:27:43,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:27:43,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166328 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:27:48,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:27:48,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171467 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:27:53,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:27:53,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812613 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:27:58,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:27:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873379 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:28:03,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:28:03,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882027 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:28:08,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:28:08,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081528 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:28:13,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:28:13,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136293 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:28:18,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:28:18,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915997 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:28:23,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:28:23,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012747 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:28:28,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:28:28,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181911 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13864::WARNING::2013-03-17 06:28:32,958::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:28:32,960::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:28:33,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:28:33,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797091 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:28:38,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:28:38,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746348 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:28:42,962::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:28:42,964::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:28:43,385::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:28:43,392::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:28:43,394::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:28:43,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:28:43,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136953 s, 299 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:28:48,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:28:48,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234501 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:28:53,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:28:53,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00067772 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:28:58,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:28:59,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832751 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:29:03,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:03,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838653 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:29:09,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:29:09,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141785 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:29:13,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:13,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208487 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:29:19,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:29:19,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076863 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:29:23,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:23,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144112 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:29:29,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:29:29,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00419947 s, 975 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:29:33,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:33,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014384 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:29:39,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:29:39,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135122 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13865::WARNING::2013-03-17 06:29:43,459::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:29:43,464::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:29:43,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:43,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185216 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:29:49,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:29:49,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793126 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:29:53,466::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:29:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:53,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:29:53,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130427 s, 314 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:29:53,944::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:29:53,950::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:29:53,951::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:29:59,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:29:59,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862906 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:30:03,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:30:03,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874157 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:30:09,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:30:09,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101546 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:30:13,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:30:13,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140363 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:30:19,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:30:19,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122522 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:30:23,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:30:23,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109083 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:30:29,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:30:29,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154584 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:30:33,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:30:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140352 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:30:39,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:30:39,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790624 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:30:43,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:30:43,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913499 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:30:49,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:30:49,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136275 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13866::WARNING::2013-03-17 06:30:53,978::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:30:53,981::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:30:53,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:30:54,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787067 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:00,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:00,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852372 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:31:03,983::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:31:03,984::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:31:03,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:04,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:04,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102917 s, 398 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:31:04,403::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:31:04,409::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:31:04,411::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:10,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:10,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143754 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:14,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:14,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843272 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:20,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:20,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156272 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:24,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:24,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00631577 s, 649 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:30,319::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:31:30,320::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:31:30,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:30,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00080406 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:30,363::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00080406 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:31:30,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:31:30,364::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:31:30,365::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:31:30,366::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:30,367::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:30,368::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:30,368::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:30,368::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:30,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:30,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:30,816::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:31:30,817::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:31:30,820::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:31:30,821::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:31:30,821::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:31:30,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:31,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:31,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:31,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183883 s, 223 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:34,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:34,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0044356 s, 923 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:41,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:41,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083368 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:44,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138824 s, 295 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:31:51,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:31:51,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250242 s, 164 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:54,264::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:31:54,269::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:31:54,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:54,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00115134 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:54,305::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00115134 s, 1.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:31:54,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:31:54,305::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:31:54,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:31:54,308::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:31:54,309::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:31:54,309::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:31:54,310::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:31:54,310::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:31:54,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:54,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:54,752::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:31:54,752::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:31:54,755::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:31:54,756::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:31:54,756::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:31:54,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:55,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:31:55,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:31:55,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142063 s, 288 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:32:01,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:32:01,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139257 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13867::WARNING::2013-03-17 06:32:04,439::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:32:04,444::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:32:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:32:05,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885057 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:32:11,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:32:11,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828684 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:32:14,447::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:32:14,449::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:32:14,881::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:32:14,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:32:14,889::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:32:15,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:32:15,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151394 s, 271 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:32:21,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:32:21,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707395 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:32:25,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:32:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769451 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:32:31,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:32:31,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813723 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:32:35,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:32:35,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167131 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:32:41,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:32:41,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771973 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:32:45,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:32:45,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155765 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:32:51,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:32:52,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148059 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:32:55,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:32:55,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728162 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:33:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:33:02,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169658 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:33:05,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:33:05,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144513 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:33:12,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:33:12,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153038 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13868::WARNING::2013-03-17 06:33:14,917::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:33:14,919::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:33:15,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:33:15,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906692 s, 452 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:33:22,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:33:22,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962492 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:33:24,922::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:33:24,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:33:25,334::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:33:25,340::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:33:25,342::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:33:25,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:33:25,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116086 s, 353 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:33:32,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:33:32,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933093 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:33:35,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:33:35,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00621691 s, 659 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:33:42,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:33:42,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200627 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:33:45,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:33:45,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158416 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:33:52,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:33:52,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015581 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:33:55,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:33:55,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113018 s, 362 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:34:02,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:34:02,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817191 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:34:05,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:34:05,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136862 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:34:12,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:34:12,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149398 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:34:15,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:34:15,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135953 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:34:22,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:34:22,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110532 s, 3.7 MB/s\n'; <rc> = 0 >Thread-13869::WARNING::2013-03-17 06:34:25,389::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:34:25,394::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:34:25,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:34:25,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732116 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:34:32,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:34:32,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831133 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:34:35,405::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:34:35,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:34:35,814::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:34:35,820::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:34:35,822::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:34:35,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:34:35,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309715 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:34:42,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:34:42,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142915 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:34:45,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:34:45,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851672 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:34:52,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:34:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785774 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:34:55,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:34:55,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800059 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:35:02,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:35:02,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037988 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:35:05,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:06,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146098 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:35:13,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:35:13,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836948 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:35:16,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:16,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322906 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:35:23,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:35:23,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108843 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:35:26,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:26,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151832 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:35:33,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:35:33,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028731 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13870::WARNING::2013-03-17 06:35:35,880::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:35:35,883::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:35:36,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:36,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083172 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:35:43,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:35:43,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784978 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:35:45,897::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:35:45,898::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:46,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:46,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140728 s, 291 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:35:46,375::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:35:46,381::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:35:46,382::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:35:53,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:35:53,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753709 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:35:56,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:35:56,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926065 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:03,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:03,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156465 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:06,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:06,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846089 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:13,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940907 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:16,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916522 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:23,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:23,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000666552 s, 6.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:26,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:26,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156068 s, 262 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:33,691::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:36:33,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:36:33,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:33,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00150073 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:33,735::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00150073 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:36:33,736::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:36:33,736::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:36:33,738::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:36:33,743::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:36:33,744::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:36:33,744::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:36:33,745::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:36:33,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:36:33,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:34,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:34,182::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:36:34,183::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:36:34,185::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:36:34,187::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:36:34,187::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:36:34,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:34,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:34,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0068597 s, 597 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:36,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:36,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00707071 s, 579 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:44,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:44,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132901 s, 308 kB/s\n'; <rc> = 0 >Thread-13871::WARNING::2013-03-17 06:36:46,410::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:36:46,413::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:36:46,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:46,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922585 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:36:55,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:36:55,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870146 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:36:56,415::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:36:56,416::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:36:56,417::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:56,455::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:36:56,458::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:36:56,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:56,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0269061 s, 76.1 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:56,520::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0269061 s, 76.1 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:36:56,520::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:36:56,521::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:36:56,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:36:56,523::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-21::DEBUG::2013-03-17 06:36:56,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:36:56,995::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:36:56,996::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,003::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,006::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,006::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,007::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:57,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:57,392::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:36:57,393::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:36:57,395::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:36:57,396::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:36:57,397::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:36:57,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:57,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:36:57,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:36:57,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187433 s, 219 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:37:05,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:37:05,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868963 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:37:07,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:37:07,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0285496 s, 143 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:37:15,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:37:15,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918405 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:37:17,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:37:17,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169207 s, 242 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:37:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:37:25,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787563 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:37:27,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:37:27,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151688 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:37:35,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:37:35,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209529 s, 195 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:37:37,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:37:38,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0326998 s, 125 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:37:45,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:37:45,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142588 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:37:48,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:37:48,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895392 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:37:55,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:37:55,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113123 s, 362 kB/s\n'; <rc> = 0 >Thread-13872::WARNING::2013-03-17 06:37:57,059::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:37:57,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:37:58,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:37:58,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148542 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:38:05,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:38:05,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157383 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:38:07,068::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:38:07,070::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:38:07,484::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:38:07,491::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:38:07,492::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:38:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:38:08,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178083 s, 230 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:38:15,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:38:16,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952474 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:38:18,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:38:18,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011014 s, 372 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:38:26,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:38:26,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865317 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:38:28,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:38:28,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084128 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:38:36,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:38:36,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770715 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:38:38,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:38:38,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154755 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:38:46,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:38:46,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206976 s, 198 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:38:48,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:38:48,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094547 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:38:56,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:38:56,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140908 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:38:58,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:38:58,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163716 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:39:06,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:39:06,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001744 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13873::WARNING::2013-03-17 06:39:07,520::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:39:07,526::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:39:08,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:39:08,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934577 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:39:16,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:39:16,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964907 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:39:17,530::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:39:17,532::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:39:17,958::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:39:17,965::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:39:17,966::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:39:18,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:39:18,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177929 s, 230 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:39:26,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:39:26,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018564 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:39:28,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:39:28,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764457 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:39:36,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:39:36,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290657 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:39:38,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:39:38,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977625 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:39:46,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:39:46,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143107 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:39:48,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:39:48,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808791 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:39:56,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:39:56,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810041 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:39:58,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:39:58,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130019 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:40:06,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:40:06,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791866 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:40:08,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:40:08,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148009 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:40:16,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:40:16,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013502 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13874::WARNING::2013-03-17 06:40:17,994::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:40:18,005::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:40:18,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:40:18,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795658 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:40:27,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:40:27,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175813 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:40:28,008::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:40:28,010::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:40:28,460::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:40:28,467::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:40:28,469::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:40:28,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:40:28,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161944 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:40:37,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:40:37,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812692 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:40:38,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:40:38,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154035 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:40:47,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:40:47,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932576 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:40:48,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:40:48,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130375 s, 314 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:40:57,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:40:57,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814219 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:40:58,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:40:58,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119454 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:07,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:07,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790463 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:41:08,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:08,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120112 s, 341 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:17,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:17,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157456 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:41:19,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:19,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886733 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:27,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:27,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160787 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13875::WARNING::2013-03-17 06:41:28,512::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:41:28,514::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:41:29,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:29,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143535 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:37,860::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:41:37,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:41:37,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:37,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0113599 s, 180 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:37,921::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0113599 s, 180 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:41:37,921::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:41:37,922::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:41:37,924::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:41:37,925::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:41:37,925::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:41:37,926::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:41:37,926::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:41:37,927::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:41:37,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:38,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:38,320::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:41:38,321::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:41:38,324::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:41:38,325::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:41:38,326::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:41:38,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-21::DEBUG::2013-03-17 06:41:38,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:41:38,521::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:38,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:38,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109772 s, 373 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:41:39,051::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:41:39,057::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:41:39,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:39,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:39,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145076 s, 282 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:48,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:48,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886736 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:41:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:49,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00441631 s, 927 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:41:58,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:41:58,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924843 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:41:59,205::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:41:59,207::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:41:59,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:59,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000962105 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:41:59,238::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000962105 s, 2.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:41:59,239::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:41:59,239::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:41:59,244::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:41:59,245::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:59,245::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:59,246::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:59,246::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:59,247::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:59,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:41:59,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:41:59,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:41:59,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:41:59,653::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:41:59,654::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:41:59,654::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:41:59,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:00,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:42:00,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:00,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911339 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:42:08,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:08,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739026 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:42:10,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:10,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083951 s, 488 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:42:19,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:19,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764565 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:42:20,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:20,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151655 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:42:29,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:29,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166006 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:42:30,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:30,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00567193 s, 722 kB/s\n'; <rc> = 0 >Thread-13876::WARNING::2013-03-17 06:42:39,084::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:42:39,091::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:42:39,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:39,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799128 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:42:40,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:40,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814656 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:42:49,093::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:42:49,095::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:42:49,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:49,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:49,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108062 s, 379 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:42:49,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:42:49,581::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:42:49,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:42:50,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:42:50,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177637 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:42:59,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:42:59,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829647 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:43:00,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:43:00,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026355 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:43:09,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:09,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893894 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:43:10,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:43:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107988 s, 379 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:43:19,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:19,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082582 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:43:20,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:43:20,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113138 s, 362 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:43:29,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:29,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820217 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:43:30,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:43:30,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087497 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:43:39,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:39,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080464 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:43:40,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:43:40,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0589295 s, 69.5 kB/s\n'; <rc> = 0 >Thread-13877::WARNING::2013-03-17 06:43:49,612::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:43:49,617::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:43:49,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:49,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017936 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:43:50,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:43:50,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823559 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:43:59,619::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:43:59,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:59,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:43:59,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167435 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:44:00,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:44:00,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:44:00,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:44:00,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:44:00,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124681 s, 329 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:44:09,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:44:09,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008272 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:44:10,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:44:10,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146062 s, 280 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:44:19,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:44:20,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154414 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:44:20,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:44:20,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134182 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:44:30,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:44:30,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137706 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:44:30,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:44:30,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890087 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:44:40,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:44:40,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154905 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:44:40,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:44:40,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832435 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:44:50,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:44:50,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087821 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:44:50,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:44:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136141 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13878::WARNING::2013-03-17 06:45:00,099::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:45:00,102::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:45:00,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:00,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284633 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:45:00,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:45:00,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869458 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:45:10,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:45:10,109::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:10,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:10,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110415 s, 3.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:45:10,571::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:45:10,577::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:45:10,578::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:45:10,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:45:10,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173236 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:45:20,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:20,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008642 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:45:20,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:45:20,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074283 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:45:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:30,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133894 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:45:30,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:45:31,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137371 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:45:40,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:40,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804097 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:45:41,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:45:41,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960659 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:45:50,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:45:50,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127357 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:45:51,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:45:51,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159175 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:00,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:00,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808713 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:46:01,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:01,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148287 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13879::WARNING::2013-03-17 06:46:10,606::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:46:10,611::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:46:10,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:10,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161036 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:46:11,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:11,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149094 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:46:20,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:46:20,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:20,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:20,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764548 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:46:21,075::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:46:21,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:46:21,081::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:46:21,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:21,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012516 s, 327 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:30,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:31,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014599 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:46:31,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:31,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171956 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:41,061::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:46:41,063::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:46:41,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:41,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00141021 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:41,104::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00141021 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:46:41,105::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:46:41,105::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:46:41,107::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:46:41,108::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:46:41,108::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:46:41,109::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:46:41,109::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:46:41,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:46:41,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:41,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:41,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369989 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:41,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:41,580::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:46:41,581::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:46:41,583::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:46:41,584::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:46:41,584::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:46:41,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:42,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:42,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143541 s, 285 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:46:51,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:46:51,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958041 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:46:52,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:46:52,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865633 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:01,466::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:47:01,467::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:47:01,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:01,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145062 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:01,518::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145062 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:47:01,518::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:47:01,518::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:47:01,520::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:47:01,521::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:47:01,521::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:47:01,522::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:47:01,522::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:47:01,522::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:47:01,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:01,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:01,939::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:47:01,940::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:47:01,942::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:47:01,943::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:47:01,944::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:47:01,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:02,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:02,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139992 s, 293 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:02,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:02,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:02,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142104 s, 288 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:47:12,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:12,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080074 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:12,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:12,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850087 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13880::WARNING::2013-03-17 06:47:21,107::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:47:21,111::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 06:47:22,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:22,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884461 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:22,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:22,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907616 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:47:31,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:47:31,115::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:47:31,594::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:47:31,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:47:31,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:47:32,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:32,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131116 s, 312 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:32,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:32,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173681 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:42,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:42,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894946 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:47:42,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113139 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:47:52,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:47:52,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892939 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:47:52,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:47:52,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939938 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:48:02,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:48:02,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847989 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:48:02,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:48:02,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853169 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:48:12,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:48:12,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151976 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:48:12,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:48:12,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00315455 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:48:22,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:48:22,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017744 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:48:22,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:48:23,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143578 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13881::WARNING::2013-03-17 06:48:31,644::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:48:31,652::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:48:32,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:48:32,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102027 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:48:33,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:48:33,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855166 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:48:41,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:48:41,656::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:48:41,657::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:48:42,091::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:48:42,098::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:48:42,100::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:48:42,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:48:42,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168951 s, 242 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:48:43,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:48:43,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144583 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:48:52,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:48:53,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909721 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:48:53,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:48:53,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811897 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:49:03,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:49:03,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153491 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:49:03,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:49:03,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809001 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:49:13,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:49:13,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809014 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:49:13,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:49:13,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936498 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:49:23,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:49:23,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848369 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:49:23,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:49:23,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820569 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:49:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:49:33,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842436 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:49:33,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:49:33,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015011 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13882::WARNING::2013-03-17 06:49:42,126::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:49:42,130::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:49:43,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:49:43,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779838 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:49:43,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:49:43,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853437 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:49:52,132::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:49:52,133::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:49:52,585::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:49:52,591::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:49:52,593::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:49:53,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:49:53,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158831 s, 258 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:49:53,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:49:53,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173941 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:50:03,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:50:03,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161427 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:50:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:50:03,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150924 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:50:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:50:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139926 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:50:13,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:50:13,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737886 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:50:23,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:50:23,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855969 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:50:23,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:50:23,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152488 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:50:33,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:50:33,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141557 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:50:33,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:50:34,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153045 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:50:43,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:50:43,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165413 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:50:44,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:50:44,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015305 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13883::WARNING::2013-03-17 06:50:52,636::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:50:52,640::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:50:53,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:50:53,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831438 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:50:54,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:50:54,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175388 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:51:02,642::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:51:02,643::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:51:03,117::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:51:03,124::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:51:03,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:51:03,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:51:03,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177522 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:04,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:04,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015398 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:51:13,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:51:13,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800802 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:14,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:14,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874385 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:51:23,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:51:23,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148921 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:24,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:24,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791979 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:51:33,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:51:33,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00510222 s, 803 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:34,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:34,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125668 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:51:43,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:51:43,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152835 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:44,522::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:51:44,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:51:44,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:44,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00906912 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:44,563::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00906912 s, 226 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:51:44,564::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:51:44,564::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:51:44,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:51:44,566::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:51:44,567::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:51:44,568::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:51:44,568::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:51:44,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:51:44,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:45,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:45,011::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:51:45,012::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:51:45,014::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:51:45,015::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:51:45,016::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:51:45,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:45,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:45,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:45,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106422 s, 385 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:51:53,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:51:53,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828986 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:51:55,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:51:55,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916335 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13884::WARNING::2013-03-17 06:52:03,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:52:03,155::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:52:03,775::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:52:03,776::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:52:03,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:03,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129085 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:03,818::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129085 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:52:03,818::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:52:03,818::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:52:03,820::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:52:03,821::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:03,822::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:03,822::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:03,822::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:03,823::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:03,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:04,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:04,273::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:04,273::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:52:04,276::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:52:04,277::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:52:04,278::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:52:04,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:04,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:04,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:04,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174412 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:52:05,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:52:05,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177488 s, 231 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:52:13,160::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:52:13,161::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:52:13,587::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:52:13,594::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:52:13,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:52:14,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:14,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164557 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:52:15,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:52:15,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871114 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:24,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:24,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135638 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:52:25,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:52:25,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708111 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:34,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:34,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819744 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:52:35,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:52:35,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967441 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:44,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:44,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754159 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:52:45,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:52:45,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738369 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:52:54,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:52:54,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152528 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:52:55,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:52:55,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222159 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:53:04,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:53:05,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157125 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:53:05,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:53:06,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777896 s, 5.3 MB/s\n'; <rc> = 0 >Thread-13885::WARNING::2013-03-17 06:53:13,624::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:53:13,630::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:53:15,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:53:15,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850719 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:53:16,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:53:16,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757596 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:53:23,633::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:53:23,634::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:53:24,161::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:53:24,168::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:53:24,169::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:53:25,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:53:25,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236826 s, 173 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:53:26,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:53:26,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768906 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:53:35,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:53:35,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172828 s, 237 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:53:36,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:53:36,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880291 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:53:45,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:53:45,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767461 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:53:46,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:53:46,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858481 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:53:55,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:53:55,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092455 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:53:56,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:53:56,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734829 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:54:05,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:54:05,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139878 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:54:06,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:54:06,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156271 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:54:15,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:54:15,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807026 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:54:16,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:54:16,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836679 s, 4.9 MB/s\n'; <rc> = 0 >Thread-13886::WARNING::2013-03-17 06:54:24,195::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:54:24,199::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:54:25,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:54:25,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869907 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:54:26,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:54:26,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840826 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:54:34,200::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 06:54:34,204::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:54:34,205::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:54:34,664::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:54:34,671::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:54:34,672::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:54:35,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:54:35,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123713 s, 331 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:54:36,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:54:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947085 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:54:45,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:54:45,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155377 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:54:46,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:54:46,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001079 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:54:55,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:54:55,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835942 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:54:56,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:54:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014694 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:55:05,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:55:05,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153436 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:55:06,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:55:06,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871629 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:55:15,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:55:15,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113586 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:55:16,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:55:16,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908212 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:55:25,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:55:25,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959011 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:55:26,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:55:27,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016368 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13887::WARNING::2013-03-17 06:55:34,697::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:55:34,701::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:55:35,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:55:35,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139752 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:55:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:55:37,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00560931 s, 730 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:55:44,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:55:44,704::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:55:45,126::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:55:45,138::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:55:45,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:55:45,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:55:45,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00932523 s, 439 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:55:47,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:55:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148961 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:55:55,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:55:55,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094612 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:55:57,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:55:57,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076827 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:56:05,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:56:05,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832128 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:07,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:07,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139055 s, 295 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:56:15,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:56:15,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152627 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:17,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:17,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942497 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:56:25,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:56:25,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821136 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:27,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716444 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:56:35,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:56:35,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090252 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:37,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306137 s, 134 kB/s\n'; <rc> = 0 >Thread-13888::WARNING::2013-03-17 06:56:45,167::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:56:45,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:56:45,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:56:45,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133597 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:47,651::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 06:56:47,653::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:56:47,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:47,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144754 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:47,690::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144754 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 06:56:47,690::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 06:56:47,690::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 06:56:47,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 06:56:47,693::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:56:47,694::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:56:47,694::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:56:47,694::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:56:47,695::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 06:56:47,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:48,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:48,126::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 06:56:48,127::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 06:56:48,129::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 06:56:48,130::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 06:56:48,131::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 06:56:48,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:48,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:48,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:48,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132863 s, 308 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:56:55,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:56:55,176::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:56:55,574::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:56:55,581::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:56:55,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:56:55,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:56:56,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129643 s, 316 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:56:58,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:56:58,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155573 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:06,022::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 06:57:06,023::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:57:06,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:06,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000832202 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:06,076::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000832202 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 06:57:06,076::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 06:57:06,077::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 06:57:06,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 06:57:06,080::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:57:06,081::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:57:06,082::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:57:06,082::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:57:06,083::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 06:57:06,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:06,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:06,472::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:57:06,473::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 06:57:06,475::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 06:57:06,477::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 06:57:06,477::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 06:57:06,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:06,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:06,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:06,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118461 s, 346 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:57:08,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:57:08,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197171 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:16,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:17,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152299 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:57:18,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:57:18,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138214 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:27,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:27,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021867 s, 187 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:57:28,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:57:28,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120295 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:37,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:37,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785797 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:57:38,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:57:38,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081702 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:57:47,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789221 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:57:48,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:57:49,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013831 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13889::WARNING::2013-03-17 06:57:55,612::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:57:55,615::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:57:57,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:57:57,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111624 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:57:59,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:57:59,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105936 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:58:05,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:58:05,619::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:58:06,017::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:58:06,024::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:58:06,025::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:58:07,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:58:07,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017945 s, 228 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:58:09,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:58:09,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847836 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:58:17,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:58:17,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146551 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:58:19,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:58:19,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880407 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:58:27,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:58:27,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079632 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:58:29,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:58:29,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009132 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:58:37,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:58:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847411 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:58:39,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:58:39,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157973 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:58:47,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:58:47,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136398 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:58:49,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:58:49,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761228 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:58:57,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:58:57,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015388 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:58:59,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:58:59,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166258 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13890::WARNING::2013-03-17 06:59:06,052::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 06:59:06,056::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 06:59:07,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:59:07,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956673 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:59:09,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:59:09,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166241 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 06:59:16,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 06:59:16,059::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 06:59:16,494::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 06:59:16,503::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 06:59:16,504::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 06:59:17,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:59:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147876 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:59:19,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:59:19,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149379 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:59:27,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:59:27,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081438 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:59:29,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:59:29,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853341 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:59:37,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:59:37,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945023 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:59:39,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:59:39,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178981 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:59:47,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:59:47,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965529 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:59:49,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 06:59:49,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148867 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 06:59:57,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 06:59:57,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151551 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 06:59:59,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:00:00,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142817 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:00:07,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:00:07,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143639 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:00:10,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:00:10,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152932 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13891::WARNING::2013-03-17 07:00:16,534::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:00:16,539::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:00:17,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:00:17,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860904 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:00:20,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:00:20,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120453 s, 3.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:00:26,541::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:00:26,542::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:00:26,544::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:00:26,973::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:00:26,980::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:00:26,981::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:00:27,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:00:28,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126913 s, 323 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:00:30,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:00:30,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907212 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:00:38,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:00:38,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776374 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:00:40,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:00:40,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109387 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:00:48,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:00:48,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109763 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:00:50,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:00:50,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149772 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:00:58,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:00:58,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147006 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:00,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:00,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848808 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:01:08,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:01:08,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883916 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:10,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:10,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017056 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:01:18,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:01:18,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747212 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:20,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:20,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231868 s, 1.8 MB/s\n'; <rc> = 0 >Thread-13892::WARNING::2013-03-17 07:01:27,010::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:01:27,016::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:01:28,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:01:28,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777148 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:30,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:30,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016121 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:01:37,018::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:01:37,019::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:01:37,444::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:01:37,450::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:01:37,452::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:01:38,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:01:38,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153881 s, 266 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:40,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:40,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0396458 s, 103 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:01:48,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:01:48,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153977 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:50,891::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:01:50,892::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:01:50,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:50,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148289 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:50,931::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148289 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:01:50,932::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:01:50,932::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:01:50,934::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:01:50,935::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:01:50,936::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:01:50,937::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:01:50,937::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:01:50,938::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:01:50,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:51,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:51,345::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:01:51,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:01:51,348::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:01:51,349::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:01:51,350::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:01:51,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:51,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:01:51,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:01:51,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00911736 s, 449 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:01:58,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:01:58,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00496736 s, 825 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:02:01,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:02:01,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820978 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:08,405::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:02:08,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:02:08,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:08,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138764 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:08,446::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138764 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:02:08,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:02:08,447::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:02:08,449::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:02:08,450::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:08,451::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:08,451::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:08,452::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:08,452::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:08,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:08,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:08,898::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:08,900::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:02:08,902::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:02:08,904::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:02:08,904::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:02:08,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:09,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:09,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:09,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194306 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:02:11,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:02:11,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835963 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:19,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:19,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162501 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:02:21,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:02:21,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828916 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:29,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143538 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:02:32,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:02:32,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972167 s, 4.2 MB/s\n'; <rc> = 0 >Thread-13893::WARNING::2013-03-17 07:02:37,477::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:02:37,481::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:02:39,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:39,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842394 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:02:42,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:02:42,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853052 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:02:47,483::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:02:47,484::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:02:47,915::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:02:47,921::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:02:47,923::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:02:49,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:49,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163365 s, 251 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:02:52,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:02:52,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170343 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:02:59,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:02:59,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132401 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:03:02,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:03:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000672072 s, 6.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:03:09,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:03:09,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837862 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:03:12,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:03:12,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879662 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:03:19,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:03:19,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158619 s, 258 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:03:22,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:03:22,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140074 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:03:29,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:03:29,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030561 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:03:32,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:03:32,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923021 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:03:39,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:03:39,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848098 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:03:42,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:03:42,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139441 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13894::WARNING::2013-03-17 07:03:47,948::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:03:47,952::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:03:49,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:03:49,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789996 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:03:52,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:03:52,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102064 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:03:57,954::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:03:57,955::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:03:58,386::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:03:58,393::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:03:58,394::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:03:59,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:03:59,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137123 s, 299 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:04:02,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:04:02,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851147 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:04:09,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:04:09,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161382 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:04:12,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:04:12,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140377 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:04:19,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:04:19,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812504 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:04:22,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:04:22,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093991 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:04:29,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:04:29,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014179 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:04:32,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:04:32,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824764 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:04:39,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:04:40,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149065 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:04:43,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:04:43,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139335 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:04:50,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:04:50,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763561 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:04:53,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:04:53,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143152 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13895::WARNING::2013-03-17 07:04:58,426::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:04:58,431::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:05:00,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:05:00,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915061 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:05:03,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:05:03,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756785 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:05:08,434::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:05:08,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:05:08,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:05:08,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:05:08,865::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:05:10,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:05:10,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207612 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:05:13,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:05:13,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150883 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:05:20,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:05:20,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942753 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:05:23,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:05:23,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925135 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:05:30,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:05:30,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140118 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:05:33,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:05:33,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128233 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:05:40,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:05:40,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152295 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:05:43,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:05:43,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861005 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:05:50,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:05:50,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977394 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:05:53,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:05:53,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134458 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:06:00,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:06:00,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142123 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:03,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:03,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151165 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13896::WARNING::2013-03-17 07:06:08,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:06:08,903::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:06:10,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:06:10,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180508 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:13,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:13,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705855 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:06:18,904::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:06:18,906::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:06:18,907::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:06:19,340::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:06:19,347::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:06:19,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:06:20,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:06:20,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121085 s, 338 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:23,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:23,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871828 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:06:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:06:30,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080172 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:33,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:33,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151629 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:06:40,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:06:40,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869589 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:43,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:44,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140137 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:06:50,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:06:50,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151881 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:54,053::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:06:54,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:06:54,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:54,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107006 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:54,097::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107006 s, 1.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:06:54,098::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:06:54,098::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:06:54,099::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:06:54,100::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:06:54,101::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:06:54,102::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:06:54,102::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:06:54,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:06:54,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:54,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:54,553::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:06:54,554::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:06:54,557::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:06:54,558::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:06:54,558::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:06:54,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:54,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:06:54,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:06:54,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00727541 s, 563 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:00,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:00,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811616 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:07:05,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:07:05,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173249 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:10,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:07:10,655::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:07:10,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:10,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140941 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:10,698::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140941 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:07:10,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:07:10,698::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:07:10,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:07:10,701::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:10,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:10,702::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:10,702::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:10,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:10,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:11,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:11,150::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:11,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:07:11,155::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:07:11,156::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:07:11,156::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:07:11,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:11,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:11,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:11,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172979 s, 237 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:07:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:07:15,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125056 s, 3.3 MB/s\n'; <rc> = 0 >Thread-13897::WARNING::2013-03-17 07:07:19,375::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:07:19,381::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:07:21,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:21,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142005 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:07:25,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:07:25,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852397 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:07:29,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:07:29,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:07:29,813::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:07:29,820::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:07:29,821::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:07:31,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:31,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00296826 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:07:35,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:07:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806288 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:41,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:41,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010288 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:07:45,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:07:45,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814249 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:07:51,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:07:51,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153232 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:07:55,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:07:55,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168917 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:08:01,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:08:01,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172755 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:08:05,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:08:05,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771192 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:08:11,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:08:11,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014208 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:08:15,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:08:15,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850828 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:08:21,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:08:21,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811997 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:08:25,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:08:25,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148462 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13898::WARNING::2013-03-17 07:08:29,847::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:08:29,850::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:08:31,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:08:32,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109618 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:08:35,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:08:35,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00997285 s, 411 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:08:39,853::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:08:39,854::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:08:40,280::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:08:40,287::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:08:40,288::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:08:42,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:08:42,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139792 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:08:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:08:45,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764614 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:08:52,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:08:52,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913653 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:08:55,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:08:55,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104974 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:09:02,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:09:02,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014028 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:09:05,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:09:05,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147227 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:09:12,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:09:12,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142753 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:09:16,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:09:16,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802319 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:09:22,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:09:22,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070924 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:09:26,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:09:26,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151968 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:09:32,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:09:32,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886534 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:09:36,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:09:36,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757225 s, 5.4 MB/s\n'; <rc> = 0 >Thread-13899::WARNING::2013-03-17 07:09:40,314::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:09:40,319::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:09:42,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:09:42,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012178 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:09:46,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:09:46,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096797 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:09:50,323::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:09:50,324::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:09:50,794::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:09:50,801::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:09:50,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:09:52,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:09:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129048 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:09:56,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:09:56,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821144 s, 499 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:10:02,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:10:02,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138242 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:10:06,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:10:06,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973137 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:10:12,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:10:12,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872679 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:10:16,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:10:16,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0025313 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:10:22,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:10:22,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142668 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:10:26,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:10:26,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144806 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:10:32,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:10:32,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0332076 s, 123 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:10:36,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:10:36,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012356 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:10:42,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:10:42,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136338 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:10:46,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:10:46,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736179 s, 5.6 MB/s\n'; <rc> = 0 >Thread-13900::WARNING::2013-03-17 07:10:50,832::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:10:50,841::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:10:52,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:10:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854979 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:10:56,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:10:56,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765478 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:11:00,844::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:11:00,845::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:11:01,250::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:11:01,257::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:11:01,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:11:02,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:11:02,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137676 s, 298 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:06,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:06,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992327 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:11:12,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:11:12,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019856 s, 206 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:16,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:16,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168562 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:11:22,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:11:22,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975376 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:27,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:27,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219978 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:11:32,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:11:32,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158048 s, 259 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:37,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:37,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817826 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:11:43,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:11:43,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139603 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:47,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:47,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798142 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:11:53,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:11:53,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144007 s, 284 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:57,294::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:11:57,304::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:11:57,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000827203 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:57,339::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000827203 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:11:57,340::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:11:57,340::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:11:57,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:11:57,344::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:11:57,345::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:11:57,345::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:11:57,345::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:11:57,346::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:11:57,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:57,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:57,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:11:57,806::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:11:57,808::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:11:57,810::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:11:57,810::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:11:57,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:58,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:11:58,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:11:58,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00450186 s, 910 kB/s\n'; <rc> = 0 >Thread-13901::WARNING::2013-03-17 07:12:01,291::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:12:01,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:12:03,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:03,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0355499 s, 115 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:12:08,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:12:08,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0284867 s, 144 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:12:11,296::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:12:11,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:12:11,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:12:11,776::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:12:11,783::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:12:11,784::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,179::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:12:13,182::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:12:13,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:13,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0135269 s, 151 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:13,223::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0135269 s, 151 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:12:13,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:12:13,224::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:12:13,227::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:12:13,228::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,229::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,230::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,230::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,230::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:13,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:13,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:12:13,651::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:12:13,653::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:12:13,655::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:12:13,655::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:12:13,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:14,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:14,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:14,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968356 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:12:18,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:12:18,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238875 s, 171 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:24,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:24,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138259 s, 296 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:12:28,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:12:28,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082756 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:34,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:34,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130099 s, 315 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:12:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:12:38,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080458 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:44,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:44,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897881 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:12:48,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:12:48,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162239 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:12:54,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:12:54,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137761 s, 297 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:12:58,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:12:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771542 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:13:04,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:13:04,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149078 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:13:08,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:13:08,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173401 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13902::WARNING::2013-03-17 07:13:11,809::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:13:11,813::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:13:14,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:13:14,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125911 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:13:18,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:13:18,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851448 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:13:21,815::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:13:21,816::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:13:22,210::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:13:22,216::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:13:22,218::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:13:24,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:13:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017748 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:13:28,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:13:28,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773705 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:13:34,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:13:34,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161139 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:13:38,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:13:39,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076184 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:13:44,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:13:44,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172725 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:13:49,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:13:49,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150369 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:13:54,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:13:54,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011335 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:13:59,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:13:59,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007165 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:14:04,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:14:04,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167675 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:14:09,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:14:09,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207756 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:14:14,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:14:14,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010598 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:14:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:14:19,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300668 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13903::WARNING::2013-03-17 07:14:22,245::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:14:22,249::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:14:24,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:14:24,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781451 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:14:29,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:14:29,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00522355 s, 784 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:14:32,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:14:32,254::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:14:32,708::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:14:32,715::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:14:32,716::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:14:34,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:14:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00711937 s, 575 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:14:39,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:14:39,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802597 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:14:44,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:14:44,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136542 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:14:49,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:14:49,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839068 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:14:54,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:14:54,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016688 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:14:59,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:14:59,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882452 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:15:04,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:15:04,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169868 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:15:09,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:15:09,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784256 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:15:14,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:15:14,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0024497 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:15:19,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:15:19,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141511 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:15:24,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:15:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769864 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:15:29,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:15:29,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853524 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13904::WARNING::2013-03-17 07:15:32,746::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:15:32,751::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:15:34,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:15:34,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00428384 s, 956 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:15:39,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:15:39,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855556 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:15:42,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:15:42,756::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:15:43,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:15:43,210::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:15:43,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:15:44,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:15:45,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00684611 s, 598 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:15:49,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:15:50,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160649 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:15:55,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:15:55,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810284 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:16:00,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:16:00,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189787 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:16:05,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:16:05,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869244 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:16:10,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:16:10,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00826973 s, 495 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:16:15,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:16:15,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890629 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:16:20,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:16:20,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806766 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:16:25,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:16:25,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735321 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:16:30,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:16:30,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014065 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:16:35,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:16:35,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206856 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:16:40,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:16:40,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112881 s, 3.6 MB/s\n'; <rc> = 0 >Thread-13905::WARNING::2013-03-17 07:16:43,237::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:16:43,240::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:16:45,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:16:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137257 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:16:50,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:16:50,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189639 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:16:53,252::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:16:53,253::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:16:53,685::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:16:53,692::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:16:53,694::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:16:55,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:16:55,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177431 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:00,580::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:17:00,581::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:17:00,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:00,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000787069 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:00,615::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000787069 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:17:00,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:17:00,616::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:17:00,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:17:00,619::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:17:00,619::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:17:00,620::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:17:00,620::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:17:00,621::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:17:00,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:01,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:01,048::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:17:01,049::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:17:01,051::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:17:01,053::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:17:01,053::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:17:01,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:01,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:01,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:01,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886076 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:05,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:05,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176362 s, 232 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:11,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:11,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077805 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:15,415::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:17:15,416::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:17:15,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:15,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819258 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:15,445::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819258 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:17:15,446::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:17:15,446::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:17:15,448::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:17:15,453::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:17:15,454::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:17:15,454::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:17:15,454::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:17:15,455::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:17:15,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:15,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:15,883::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:17:15,884::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:17:15,887::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:17:15,888::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:17:15,888::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:17:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:16,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:16,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:16,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134003 s, 306 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:21,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:21,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144529 s, 283 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:26,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:26,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082479 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:31,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:31,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076634 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:36,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:36,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088086 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:41,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:41,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143564 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:17:46,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:46,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140611 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:17:51,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:17:51,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802305 s, 5.1 MB/s\n'; <rc> = 0 >Thread-13906::WARNING::2013-03-17 07:17:53,721::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:17:53,725::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:17:56,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:17:56,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795093 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:18:02,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:18:02,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00298475 s, 1.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:18:03,726::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:18:03,728::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:18:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:18:04,143::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:18:04,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:18:04,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:18:06,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:18:06,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105506 s, 388 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:18:12,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:18:12,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00584883 s, 700 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:18:16,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:18:16,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850809 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:18:22,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:18:22,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894784 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:18:26,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:18:26,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154319 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:18:32,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:18:32,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110556 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:18:36,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:18:36,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858353 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:18:42,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:18:42,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175873 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:18:46,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:18:46,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170365 s, 240 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:18:52,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:18:52,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144377 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:18:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:18:56,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144965 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:19:02,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:19:02,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911529 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13907::WARNING::2013-03-17 07:19:04,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:19:04,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:19:06,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:19:06,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01202 s, 341 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:19:12,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:19:12,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281981 s, 1.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:19:14,183::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:19:14,185::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:19:14,590::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:19:14,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:19:14,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:19:16,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:19:16,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142892 s, 287 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:19:22,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:19:22,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137392 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:19:26,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:19:26,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00952011 s, 430 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:19:33,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:19:33,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798278 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:19:36,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:19:36,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106371 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:19:43,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:19:43,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687029 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:19:46,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:19:46,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119413 s, 343 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:19:53,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:19:53,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881434 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:19:56,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:19:57,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867317 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:20:03,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:20:03,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154436 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:20:07,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:20:07,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181673 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:20:13,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:20:13,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826293 s, 5.0 MB/s\n'; <rc> = 0 >Thread-13908::WARNING::2013-03-17 07:20:14,625::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:20:14,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:20:17,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:20:17,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805763 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:20:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:20:23,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100285 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:20:24,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:20:24,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:20:25,067::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:20:25,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:20:25,075::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:20:27,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:20:27,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114019 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:20:33,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:20:33,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739209 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:20:37,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:20:37,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135252 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:20:43,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:20:43,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125773 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:20:47,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:20:47,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127474 s, 321 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:20:53,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:20:53,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157753 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:20:57,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:20:57,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834282 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:21:03,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:21:03,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773783 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:21:07,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:21:07,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128887 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:21:13,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:21:13,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075047 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:21:17,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:21:17,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925426 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:21:23,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:21:23,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128553 s, 3.2 MB/s\n'; <rc> = 0 >Thread-13909::WARNING::2013-03-17 07:21:25,101::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:21:25,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:21:27,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:21:27,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168463 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:21:34,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:21:34,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00883633 s, 464 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:21:35,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:21:35,109::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:21:35,531::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:21:35,538::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:21:35,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:21:37,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:21:37,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132931 s, 308 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:21:44,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:21:44,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120561 s, 340 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:21:47,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:21:47,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149443 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:21:54,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:21:54,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803581 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:21:57,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:21:57,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159723 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:04,302::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:22:04,303::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:22:04,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:04,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152728 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:04,350::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152728 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:22:04,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:22:04,350::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:22:04,352::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:22:04,353::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:22:04,354::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:22:04,354::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:22:04,355::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:22:04,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:22:04,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:04,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:04,807::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:22:04,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:22:04,830::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:22:04,832::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:22:04,833::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:22:04,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:05,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:05,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:05,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104422 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:07,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:07,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199878 s, 205 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:15,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:15,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176712 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:17,650::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:22:17,652::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:22:17,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:17,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132413 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:17,691::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132413 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:22:17,694::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:22:17,695::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:22:17,696::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:22:17,697::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:17,698::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:17,698::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:17,698::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:17,699::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:17,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:18,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:18,147::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:18,148::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:22:18,150::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:22:18,152::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:22:18,152::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:22:18,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:18,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:18,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:18,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00534152 s, 767 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:25,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:25,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797227 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:28,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:28,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115234 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:35,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:35,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767441 s, 5.3 MB/s\n'; <rc> = 0 >Thread-13910::WARNING::2013-03-17 07:22:35,571::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:22:35,586::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:22:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:38,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028938 s, 1.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:22:45,589::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:22:45,591::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:45,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:45,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142697 s, 287 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:22:46,141::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:22:46,147::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:22:46,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:22:48,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:48,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010316 s, 397 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:22:55,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:22:55,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729837 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:22:58,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:22:58,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118697 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:23:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:05,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897893 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:23:08,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:23:08,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096046 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:23:15,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:15,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000692898 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:23:18,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:23:18,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842518 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:23:25,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:25,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090084 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:23:28,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:23:28,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815241 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:23:36,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:36,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151668 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:23:38,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:23:38,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136783 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:23:46,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:46,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0048428 s, 846 kB/s\n'; <rc> = 0 >Thread-13911::WARNING::2013-03-17 07:23:46,180::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:23:46,191::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:23:48,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:23:48,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117632 s, 348 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:23:56,195::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:23:56,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:23:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:56,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:23:56,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141766 s, 289 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:23:56,716::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:23:56,721::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:23:56,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:23:58,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:23:58,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00620608 s, 660 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:24:06,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:24:06,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141203 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:24:08,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:24:09,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949345 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:24:16,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:24:16,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756846 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:24:19,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:24:19,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137967 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:24:26,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:24:26,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827756 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:24:29,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:24:29,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011087 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:24:36,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:24:36,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142948 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:24:39,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:24:39,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957021 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:24:46,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:24:46,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171515 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:24:49,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:24:49,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143741 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13912::WARNING::2013-03-17 07:24:56,749::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:24:56,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:24:56,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:24:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853914 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:24:59,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:24:59,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813243 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:25:06,755::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:25:06,757::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:06,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:06,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164825 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:25:07,238::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:25:07,244::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:25:07,245::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:25:09,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:25:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136084 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:25:16,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:16,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779554 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:25:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:25:19,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979681 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:25:27,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:27,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892792 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:25:29,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:25:29,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808492 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:25:37,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:37,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172806 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:25:39,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:25:39,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162248 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:25:47,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:47,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836272 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:25:49,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:25:49,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143941 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:25:57,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:25:57,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880436 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:25:59,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:25:59,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014632 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13913::WARNING::2013-03-17 07:26:07,272::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:26:07,279::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:26:07,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:07,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927836 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:26:09,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:26:09,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985187 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:26:17,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:26:17,284::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:17,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:17,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155217 s, 264 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:26:17,817::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:26:17,822::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:26:17,823::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:26:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:26:19,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116389 s, 352 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:26:27,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:27,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816033 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:26:29,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:26:29,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835121 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:26:37,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:37,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00233324 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:26:39,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:26:39,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012927 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:26:47,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:47,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773363 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:26:49,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:26:49,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820717 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:26:57,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:26:57,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891203 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:26:59,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:26:59,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145801 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:07,813::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:27:07,814::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:27:07,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:07,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132167 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:07,852::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132167 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:27:07,853::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:27:07,853::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:27:07,855::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:27:07,859::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:07,860::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:07,861::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:07,861::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:07,861::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:07,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:08,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:08,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:08,309::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:27:08,311::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:27:08,312::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:27:08,313::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:27:08,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:08,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:08,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00641283 s, 639 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:09,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:09,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112061 s, 366 kB/s\n'; <rc> = 0 >Thread-13914::WARNING::2013-03-17 07:27:17,849::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:27:17,853::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:27:18,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:18,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799795 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:19,790::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:27:19,792::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:27:19,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:19,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00962026 s, 213 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:19,856::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00962026 s, 213 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:27:19,856::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:27:19,856::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:27:19,858::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:27:19,859::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:27:19,860::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:27:19,860::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:27:19,861::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:27:19,861::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:27:19,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:20,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:20,265::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:27:20,266::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:27:20,269::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:27:20,270::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:27:20,270::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:27:20,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:20,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:20,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:20,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173478 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:27:27,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:27:27,856::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:27:28,266::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:27:28,272::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:27:28,274::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:27:28,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:28,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144888 s, 283 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:30,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:30,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129999 s, 315 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:38,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:38,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725995 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:40,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:40,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827935 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:48,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:49,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701058 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:27:50,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:27:50,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143238 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:27:59,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:27:59,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799354 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:28:00,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:28:00,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151684 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:28:09,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:28:09,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135045 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:28:10,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:28:10,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805259 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:28:19,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:28:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171777 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:28:20,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:28:20,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00233664 s, 1.8 MB/s\n'; <rc> = 0 >Thread-13915::WARNING::2013-03-17 07:28:28,306::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:28:28,309::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:28:29,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:28:29,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819028 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:28:30,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:28:30,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781336 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:28:38,311::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:28:38,313::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:28:38,742::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:28:38,749::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:28:38,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:28:39,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:28:39,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105942 s, 387 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:28:40,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:28:41,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00885925 s, 462 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:28:49,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:28:49,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829112 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:28:51,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:28:51,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127055 s, 322 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:28:59,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:28:59,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153594 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:29:01,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:29:01,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0086487 s, 474 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:29:09,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:29:09,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897214 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:29:11,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:29:11,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014441 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:29:19,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:29:19,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924667 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:29:21,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:29:21,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141619 s, 289 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:29:29,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:29:29,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146012 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:29:31,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:29:31,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109418 s, 3.7 MB/s\n'; <rc> = 0 >Thread-13916::WARNING::2013-03-17 07:29:38,776::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:29:38,780::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:29:39,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:29:39,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853281 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:29:41,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:29:41,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143398 s, 286 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:29:48,782::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:29:48,784::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:29:48,785::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:29:49,178::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:29:49,185::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:29:49,186::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:29:49,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:29:49,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143442 s, 286 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:29:51,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:29:51,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115666 s, 354 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:29:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:30:00,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850989 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:30:01,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:30:01,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153963 s, 266 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:30:10,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:30:10,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143552 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:30:11,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:30:11,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158125 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:30:20,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:30:20,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161595 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:30:21,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:30:21,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150987 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:30:30,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:30:30,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014297 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:30:31,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:30:31,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015923 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:30:40,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:30:40,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864493 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:30:41,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:30:41,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153654 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13917::WARNING::2013-03-17 07:30:49,213::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:30:49,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:30:50,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:30:50,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919285 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:30:51,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:30:51,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748295 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:30:59,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:30:59,223::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:30:59,679::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:30:59,686::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:30:59,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:31:00,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:31:00,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164021 s, 250 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:31:01,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:31:01,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179221 s, 229 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:31:10,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:31:10,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149003 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:31:11,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:31:11,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120946 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:31:20,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:31:20,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131501 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:31:21,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:31:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238179 s, 172 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:31:30,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:31:30,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019719 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:31:31,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:31:31,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008703 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:31:40,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:31:40,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154573 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:31:41,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:31:41,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186227 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:31:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:31:50,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757925 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:31:51,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:31:52,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013298 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13918::WARNING::2013-03-17 07:31:59,717::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:31:59,731::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:32:00,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:00,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994598 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:02,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:02,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175182 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:32:09,735::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:32:09,737::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:32:10,189::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:32:10,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:32:10,197::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,004::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:32:11,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:32:11,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:11,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0149339 s, 137 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:11,052::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0149339 s, 137 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:32:11,053::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:32:11,053::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:32:11,055::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:32:11,056::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,057::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,057::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,057::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:11,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:11,437::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:32:11,438::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:32:11,441::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:32:11,442::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:32:11,443::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:32:11,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:11,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:11,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:11,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190459 s, 215 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:12,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:12,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00490294 s, 835 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:21,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:21,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075812 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:22,128::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:32:22,130::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:32:22,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:22,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819783 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:22,158::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819783 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:32:22,159::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:32:22,159::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:32:22,161::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:32:22,162::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:32:22,162::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:32:22,163::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:32:22,163::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:32:22,164::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:32:22,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:22,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:22,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:32:22,571::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:32:22,574::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:32:22,575::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:32:22,575::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:32:22,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:22,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:22,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:22,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00282268 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:31,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:31,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880359 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:32,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:33,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101367 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:42,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:42,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138493 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:43,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:43,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997137 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:32:52,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:32:52,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133298 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:32:53,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:32:53,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136814 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:33:02,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:33:02,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228123 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:33:03,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:33:03,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906442 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13919::WARNING::2013-03-17 07:33:10,225::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:33:10,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:33:12,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:33:12,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852892 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:33:13,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:33:13,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085134 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:33:20,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:33:20,232::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:33:20,717::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:33:20,724::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:33:20,725::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:33:22,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:33:22,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189495 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:33:23,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:33:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135925 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:33:32,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:33:32,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087762 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:33:33,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:33:33,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866022 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:33:42,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:33:42,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856434 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:33:43,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:33:43,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299017 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:33:52,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:33:52,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155041 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:33:53,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:33:53,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00709219 s, 578 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:34:02,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:34:02,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788558 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:34:03,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:34:03,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161593 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:34:12,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:34:12,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134853 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:34:13,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:34:13,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00297861 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13920::WARNING::2013-03-17 07:34:20,753::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:34:20,755::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:34:22,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:34:22,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794332 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:34:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:34:23,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805023 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:34:30,758::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:34:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:34:31,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:34:31,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:34:31,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:34:32,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:34:32,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899711 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:34:33,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:34:33,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167526 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:34:42,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:34:42,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170194 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:34:43,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:34:43,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153655 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:34:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:34:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864957 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:34:53,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:34:53,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148363 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:35:03,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:35:03,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018629 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:35:03,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:35:03,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228845 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:35:13,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:35:13,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156678 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:35:13,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:35:13,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847254 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:35:23,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:35:23,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111165 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:35:23,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:35:23,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015461 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13921::WARNING::2013-03-17 07:35:31,194::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:35:31,198::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:35:33,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:35:33,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265111 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:35:33,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:35:33,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925854 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:35:41,200::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:35:41,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:35:41,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:35:41,617::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:35:41,623::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:35:41,625::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:35:43,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:35:43,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747545 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:35:43,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:35:43,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127637 s, 321 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:35:53,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:35:53,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128531 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:35:53,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:35:53,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130351 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:36:03,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:36:03,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875239 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:36:03,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:36:03,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809543 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:36:13,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:36:13,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087573 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:36:13,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:36:13,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00246483 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:36:23,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:36:23,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794862 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:36:23,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:36:24,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139237 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:36:33,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:36:33,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209499 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:36:34,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:36:34,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159731 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13922::WARNING::2013-03-17 07:36:41,649::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:36:41,653::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 07:36:43,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:36:43,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922736 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:36:44,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:36:44,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140888 s, 291 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:36:51,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:36:51,670::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:36:52,067::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:36:52,074::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:36:52,075::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:36:53,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:36:53,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794267 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:36:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:36:54,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154068 s, 266 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:04,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:04,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00941402 s, 435 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:04,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:04,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886208 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:14,115::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:37:14,116::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:37:14,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:14,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000926677 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:14,156::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000926677 s, 2.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:37:14,156::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:37:14,157::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:37:14,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:37:14,160::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:37:14,161::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:37:14,162::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:37:14,162::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:37:14,162::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:37:14,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:14,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:14,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194312 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:14,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:14,609::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:37:14,610::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:37:14,612::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:37:14,613::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:37:14,613::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:37:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:15,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:15,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:15,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153967 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:24,280::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:37:24,282::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:37:24,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:24,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0099744 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:24,326::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0099744 s, 205 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:37:24,326::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:37:24,326::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:37:24,328::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:37:24,330::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:37:24,331::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:37:24,331::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:37:24,332::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:37:24,332::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:37:24,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:24,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:24,745::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:37:24,746::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:37:24,748::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:37:24,750::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:37:24,750::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:37:24,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:25,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:25,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:25,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:25,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101264 s, 404 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:25,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174362 s, 235 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:35,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:35,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:35,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890315 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:35,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117132 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:37:45,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:45,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:45,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085576 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:45,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118452 s, 3.5 MB/s\n'; <rc> = 0 >Thread-13923::WARNING::2013-03-17 07:37:52,119::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:37:52,122::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:37:55,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:37:55,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821257 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:37:55,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:37:55,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087353 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:38:02,124::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:38:02,126::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:38:02,519::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:38:02,526::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:38:02,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:38:05,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:38:05,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126087 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:38:05,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:38:05,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846276 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:38:15,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:38:15,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165493 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:38:15,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:38:15,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163707 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:38:25,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:38:25,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179193 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:38:25,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:38:25,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736229 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:38:35,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:38:35,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137578 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:38:35,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:38:35,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162565 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:38:45,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:38:45,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139548 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:38:45,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:38:45,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148669 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:38:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:38:55,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153364 s, 267 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:38:55,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:38:55,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969317 s, 4.2 MB/s\n'; <rc> = 0 >Thread-13924::WARNING::2013-03-17 07:39:02,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:39:02,558::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:39:05,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:39:05,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107371 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:39:06,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:39:06,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100647 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:39:12,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:39:12,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:39:12,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:39:12,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:39:12,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:39:15,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:39:15,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156714 s, 261 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:39:16,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:39:16,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794683 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:39:25,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:39:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127313 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:39:26,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:39:26,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792829 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:39:35,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:39:35,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168282 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:39:36,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:39:36,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158599 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:39:45,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:39:45,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169918 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:39:46,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:39:46,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155591 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:39:55,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:39:56,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127024 s, 322 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:39:56,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:39:56,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142765 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:40:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:40:06,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898887 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:40:06,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:40:06,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823693 s, 5.0 MB/s\n'; <rc> = 0 >Thread-13925::WARNING::2013-03-17 07:40:13,015::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:40:13,018::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:40:16,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:40:16,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151095 s, 271 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:40:16,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:40:16,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740624 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:40:23,021::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:40:23,022::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:40:23,455::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:40:23,462::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:40:23,464::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:40:26,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:40:26,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121553 s, 337 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:40:26,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:40:26,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164431 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:40:36,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:40:36,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924704 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:40:36,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:40:36,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169103 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:40:46,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:40:46,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879661 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:40:46,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:40:46,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258986 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:40:56,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:40:56,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171803 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:40:56,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:40:56,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738937 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:41:06,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:41:06,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134491 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:41:06,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:41:07,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147026 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:41:16,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:41:16,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202257 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:41:17,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:41:17,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133953 s, 3.1 MB/s\n'; <rc> = 0 >Thread-13926::WARNING::2013-03-17 07:41:23,489::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:41:23,493::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:41:26,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:41:26,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780527 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:41:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:41:27,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893479 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:41:33,494::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:41:33,496::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:41:33,497::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:41:33,932::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:41:33,939::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:41:33,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:41:36,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:41:36,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159221 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:41:37,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:41:37,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873541 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:41:46,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:41:46,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992703 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:41:47,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:41:47,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926451 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:41:56,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:41:56,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811297 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:41:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:41:57,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179052 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:06,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:06,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156286 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:07,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:07,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879109 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:16,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:16,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140437 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:17,485::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:42:17,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:42:17,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:17,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000767976 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:17,522::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000767976 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:42:17,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:42:17,523::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:42:17,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:42:17,525::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:42:17,526::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:42:17,526::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:42:17,527::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:42:17,527::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:42:17,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:17,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:17,956::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:42:17,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:42:17,959::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:42:17,960::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:42:17,961::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:42:17,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:18,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:18,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:18,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178963 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:26,623::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:42:26,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:42:26,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:26,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126161 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:26,667::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126161 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:42:26,667::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:42:26,668::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:42:26,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:42:26,670::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:26,671::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:26,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:26,671::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:26,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:26,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:27,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:27,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:27,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:42:27,112::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:42:27,113::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:42:27,114::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:42:27,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:27,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:27,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:27,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157203 s, 261 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:28,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:28,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182448 s, 225 kB/s\n'; <rc> = 0 >Thread-13927::WARNING::2013-03-17 07:42:33,983::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:42:33,986::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:42:37,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:37,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177189 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:38,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:38,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809524 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:42:43,989::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:42:43,990::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:42:44,438::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:42:44,445::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:42:44,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:42:47,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:47,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191049 s, 214 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:48,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:48,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887629 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:42:57,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:42:57,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834055 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:42:58,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:42:58,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148795 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:43:07,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:43:07,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827079 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:43:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:43:08,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741665 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:43:17,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:43:17,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149215 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:43:18,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:43:18,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148223 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:43:27,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:43:27,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151495 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:43:28,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:43:28,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852966 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:43:37,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:43:37,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095648 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:43:38,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:43:39,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115619 s, 3.5 MB/s\n'; <rc> = 0 >Thread-13928::WARNING::2013-03-17 07:43:44,484::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:43:44,490::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:43:47,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:43:47,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916901 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:43:49,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:43:49,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136854 s, 3.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:43:54,494::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:43:54,495::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:43:54,913::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:43:54,920::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:43:54,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:43:57,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:43:57,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111499 s, 367 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:43:59,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:43:59,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848553 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:44:07,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:44:08,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106281 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:44:09,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:44:09,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814683 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:44:18,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:44:18,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734394 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:44:19,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:44:19,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154381 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:44:28,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:44:28,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864444 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:44:29,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:44:29,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147068 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:44:38,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:44:38,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142315 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:44:39,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:44:39,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155801 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:44:48,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:44:48,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145252 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:44:49,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:44:49,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119622 s, 3.4 MB/s\n'; <rc> = 0 >Thread-13929::WARNING::2013-03-17 07:44:54,952::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:44:54,958::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:44:58,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:44:58,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022482 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:44:59,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:44:59,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781754 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:45:04,960::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:45:04,961::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:45:05,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:45:05,378::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:45:05,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:45:08,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:45:08,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016182 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:45:09,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:45:09,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867279 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:45:18,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:45:18,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107099 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:45:19,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:45:19,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743548 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:45:28,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:45:28,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742097 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:45:29,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:45:29,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838137 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:45:38,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:45:38,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852179 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:45:39,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:45:39,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145915 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:45:48,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:45:48,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077061 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:45:49,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:45:50,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734367 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:45:58,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:45:58,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00428237 s, 956 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:46:00,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:46:00,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161248 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13930::WARNING::2013-03-17 07:46:05,413::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:46:05,415::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:46:08,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:46:08,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761377 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:46:10,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:46:10,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898922 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:46:15,417::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:46:15,419::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:46:15,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:46:15,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:46:15,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:46:18,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:46:18,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184555 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:46:20,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:46:20,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837674 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:46:28,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:46:28,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0092022 s, 445 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:46:30,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:46:30,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101206 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:46:38,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:46:38,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0046672 s, 878 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:46:40,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:46:40,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114145 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:46:49,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:46:49,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000664055 s, 6.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:46:50,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:46:50,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00456976 s, 896 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:46:59,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:46:59,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795142 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:00,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:00,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134672 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:47:09,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:09,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136996 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:10,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:10,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834503 s, 4.9 MB/s\n'; <rc> = 0 >Thread-13931::WARNING::2013-03-17 07:47:15,870::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:47:15,876::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:47:19,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:19,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984983 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:20,684::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:47:20,686::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:47:20,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:20,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000869274 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:20,717::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000869274 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:47:20,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:47:20,718::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:47:20,720::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:47:20,721::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:47:20,722::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:47:20,722::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:47:20,723::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:47:20,723::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:47:20,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:21,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:21,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:47:21,152::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:47:21,154::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:47:21,155::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:47:21,156::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:47:21,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:21,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:21,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:21,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149774 s, 273 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:47:25,879::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:47:25,880::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:47:25,881::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:47:26,298::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:47:26,305::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:47:26,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,182::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:47:29,185::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:47:29,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:29,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0126819 s, 161 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:47:29,236::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0126819 s, 161 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:47:29,237::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:47:29,237::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:47:29,238::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:47:29,239::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,240::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,240::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,241::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,241::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:29,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:47:29,682::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:47:29,686::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:47:29,688::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:47:29,689::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:47:29,690::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:47:29,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:30,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:47:30,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:30,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0079532 s, 515 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:31,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:31,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117281 s, 349 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:47:40,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:40,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952805 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:41,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:41,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816775 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:47:50,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:47:50,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706538 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:47:51,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:47:51,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144603 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:48:00,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:48:00,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196649 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:48:01,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:48:01,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931608 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:48:10,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:48:10,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136739 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:48:11,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:48:11,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151913 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:48:20,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:48:20,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143982 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:48:21,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:48:22,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878798 s, 4.7 MB/s\n'; <rc> = 0 >Thread-13932::WARNING::2013-03-17 07:48:26,345::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:48:26,348::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:48:30,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:48:30,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074654 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:48:32,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:48:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114783 s, 3.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:48:36,350::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:48:36,351::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:48:36,765::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:48:36,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:48:36,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:48:40,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:48:40,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130242 s, 314 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:48:42,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:48:42,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774643 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:48:50,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:48:50,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869985 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:48:52,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:48:52,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159465 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:49:00,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:49:00,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827646 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:49:02,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:49:02,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142625 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:49:10,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:49:10,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838986 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:49:12,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:49:12,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156989 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:49:20,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:49:20,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130137 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:49:22,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:49:22,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00679779 s, 603 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:49:30,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:49:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810029 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:49:32,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:49:32,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999905 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13933::WARNING::2013-03-17 07:49:36,802::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:49:36,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:49:40,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:49:40,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111082 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:49:42,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:49:42,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826631 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:49:46,812::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:49:46,813::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:49:47,284::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:49:47,296::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:49:47,297::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:49:50,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:49:50,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149576 s, 274 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:49:52,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:49:52,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840145 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:50:00,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:50:00,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862341 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:50:02,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:50:02,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994749 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:50:10,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:50:10,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978212 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:50:12,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:50:12,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803721 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:50:20,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:50:20,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850071 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:50:22,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:50:22,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987856 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:50:30,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:50:30,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155247 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:50:32,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:50:33,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768803 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:50:40,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:50:40,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135672 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:50:43,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:50:43,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171219 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13934::WARNING::2013-03-17 07:50:47,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:50:47,330::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:50:50,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:50:51,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911405 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:50:53,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:50:53,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129263 s, 3.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:50:57,334::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:50:57,335::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:50:57,748::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:50:57,755::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:50:57,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:51:01,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:51:01,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109147 s, 375 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:51:03,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:51:03,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145455 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:51:11,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:51:11,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827572 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:51:13,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:51:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705617 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:51:21,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:51:21,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137719 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:51:23,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:51:23,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136879 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:51:31,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:51:31,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743946 s, 551 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:51:33,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:51:33,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898407 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:51:41,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:51:41,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008597 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:51:43,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:51:43,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141231 s, 290 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:51:51,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:51:51,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149691 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:51:53,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:51:53,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950788 s, 4.3 MB/s\n'; <rc> = 0 >Thread-13935::WARNING::2013-03-17 07:51:57,784::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:51:57,790::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:52:01,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:01,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834422 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:03,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845516 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:52:07,792::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:52:07,794::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:52:08,196::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:52:08,203::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:52:08,204::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:11,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102505 s, 400 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:13,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:13,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081862 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:21,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:21,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783304 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:23,973::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:52:23,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:52:23,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:24,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000888821 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:24,005::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000888821 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:52:24,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:52:24,006::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:52:24,008::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:52:24,009::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:52:24,009::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:52:24,010::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:52:24,010::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:52:24,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:52:24,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:24,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:24,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:52:24,467::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:52:24,469::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:52:24,470::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:52:24,471::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:52:24,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:24,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:24,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:24,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150309 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:31,384::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:52:31,386::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:52:31,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:31,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00759635 s, 270 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:31,421::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00759635 s, 270 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:52:31,422::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:52:31,422::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:52:31,424::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:52:31,425::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:31,425::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:31,426::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:31,426::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:31,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:31,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:31,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:31,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:52:31,806::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:52:31,808::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:52:31,809::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:52:31,810::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:52:31,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:32,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:32,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159262 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:35,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:35,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742497 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:42,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:42,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775196 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:45,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177005 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:52:52,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:52:52,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115276 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:52:55,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:52:55,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859408 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:53:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:53:02,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017478 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:53:05,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:53:05,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0097075 s, 422 kB/s\n'; <rc> = 0 >Thread-13936::WARNING::2013-03-17 07:53:08,233::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:53:08,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:53:12,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:53:12,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103567 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:53:15,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:53:15,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163513 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:53:18,243::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:53:18,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:53:18,245::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:53:18,638::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:53:18,645::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:53:18,646::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:53:22,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:53:22,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122389 s, 335 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:53:25,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:53:25,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151502 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:53:32,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:53:32,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886624 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:53:35,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:53:35,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724051 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:53:42,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:53:42,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823766 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:53:45,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:53:45,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762601 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:53:52,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:53:52,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00432389 s, 947 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:53:55,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:53:55,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162934 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:54:02,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:54:02,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289013 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:54:05,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:54:05,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863719 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:54:12,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:54:12,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114056 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:54:15,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:54:15,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144592 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13937::WARNING::2013-03-17 07:54:18,673::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:54:18,676::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:54:22,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:54:22,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135659 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:54:25,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:54:25,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124969 s, 328 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:54:28,678::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:54:28,680::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:54:29,089::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:54:29,123::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:54:29,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:54:32,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:54:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138911 s, 295 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:54:35,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:54:36,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117339 s, 349 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:54:42,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:54:42,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890472 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:54:46,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:54:46,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148993 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:54:52,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:54:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828969 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:54:56,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:54:56,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121537 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:55:02,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:55:02,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768624 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:55:06,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:55:06,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134428 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:55:12,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:55:12,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129807 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:55:16,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:55:16,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158723 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:55:22,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:55:22,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030674 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:55:26,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:55:26,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116135 s, 3.5 MB/s\n'; <rc> = 0 >Thread-13938::WARNING::2013-03-17 07:55:29,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:55:29,153::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:55:32,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:55:32,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820065 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:55:36,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:55:36,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851757 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:55:39,158::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:55:39,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:55:39,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:55:39,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:55:39,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:55:42,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:55:43,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119484 s, 343 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:55:46,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:55:46,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810859 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:55:53,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:55:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763639 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:55:56,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:55:56,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819905 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:56:03,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:56:03,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895452 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:56:06,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:56:06,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949514 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:56:13,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:56:13,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864053 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:56:16,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:56:16,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779194 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:56:23,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:56:23,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016265 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:56:26,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:56:26,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144054 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:56:33,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:56:33,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155515 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:56:36,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:56:36,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851937 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13939::WARNING::2013-03-17 07:56:39,601::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:56:39,603::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:56:43,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:56:43,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176926 s, 232 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:56:46,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:56:47,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128917 s, 318 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:56:49,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:56:49,608::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:56:50,016::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:56:50,023::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:56:50,024::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:56:53,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:56:53,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138555 s, 296 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:56:57,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:56:57,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136715 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:03,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:03,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184557 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:07,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:07,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730971 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:13,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149863 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:17,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:17,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996113 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:23,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:23,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180372 s, 227 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:27,304::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 07:57:27,306::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:57:27,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:27,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000919055 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:27,338::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000919055 s, 2.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 07:57:27,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 07:57:27,338::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 07:57:27,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 07:57:27,341::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:57:27,342::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:57:27,342::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:57:27,342::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:57:27,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 07:57:27,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:27,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:27,825::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 07:57:27,826::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 07:57:27,829::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 07:57:27,830::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 07:57:27,830::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 07:57:27,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:28,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:28,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:28,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770422 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:33,489::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 07:57:33,491::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:57:33,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:33,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0167379 s, 122 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:33,548::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0167379 s, 122 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 07:57:33,549::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 07:57:33,549::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 07:57:33,551::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 07:57:33,552::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:57:33,552::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:57:33,553::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:57:33,553::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:57:33,554::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 07:57:33,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:33,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:33,984::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:57:33,985::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 07:57:33,988::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 07:57:33,989::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 07:57:33,989::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 07:57:33,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:34,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:34,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:34,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131025 s, 313 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:38,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:38,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153133 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:57:44,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:44,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238704 s, 172 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:48,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:48,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992184 s, 413 kB/s\n'; <rc> = 0 >Thread-13940::WARNING::2013-03-17 07:57:50,050::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:57:50,055::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:57:54,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:57:54,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728848 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:57:58,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:57:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886534 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:58:00,057::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:58:00,058::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:58:00,495::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:58:00,502::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:58:00,503::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:58:04,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:58:04,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212441 s, 193 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:58:08,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:58:08,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872024 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:58:14,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:58:14,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850157 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:58:18,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:58:18,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154812 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:58:24,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:58:24,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193592 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:58:28,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:58:28,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815838 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:58:34,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:58:34,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783135 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:58:38,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:58:38,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137381 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:58:44,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:58:44,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00496427 s, 825 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:58:48,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:58:48,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075555 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:58:54,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:58:54,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013463 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:58:58,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:58:58,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795275 s, 5.2 MB/s\n'; <rc> = 0 >Thread-13941::WARNING::2013-03-17 07:59:00,530::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 07:59:00,534::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 07:59:04,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:59:04,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00946368 s, 433 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:59:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:59:09,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912299 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 07:59:10,536::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 07:59:10,537::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 07:59:10,539::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 07:59:10,930::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 07:59:10,937::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 07:59:10,939::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 07:59:14,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:59:14,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133016 s, 308 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:59:19,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:59:19,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821582 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:59:24,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:59:24,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875694 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:59:29,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:59:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158239 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:59:34,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:59:34,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771184 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:59:39,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:59:39,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135333 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:59:44,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:59:45,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016424 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:59:49,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:59:49,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152927 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 07:59:55,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 07:59:55,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569125 s, 720 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 07:59:59,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 07:59:59,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906217 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:00:05,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:00:05,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141649 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:00:09,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:00:09,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100445 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13942::WARNING::2013-03-17 08:00:10,968::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:00:10,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:00:15,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:00:15,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00235496 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:00:19,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:00:19,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907491 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:00:20,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:00:20,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:00:21,468::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:00:21,474::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:00:21,476::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:00:25,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:00:25,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00900252 s, 455 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:00:30,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:00:30,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831308 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:00:35,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:00:35,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793226 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:00:40,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:00:40,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967067 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:00:45,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:00:45,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841102 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:00:50,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:00:50,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755724 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:00:55,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:00:55,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792674 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:01:00,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:01:00,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083465 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:01:05,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:01:05,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001477 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:01:10,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:01:10,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259236 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:01:15,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:01:15,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102761 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:01:20,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:01:20,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00595139 s, 688 kB/s\n'; <rc> = 0 >Thread-13943::WARNING::2013-03-17 08:01:21,507::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:01:21,511::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:01:25,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:01:25,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879373 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:01:30,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:01:30,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805429 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:01:31,513::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:01:31,515::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:01:31,908::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:01:31,915::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:01:31,917::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:01:35,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:01:35,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111678 s, 367 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:01:40,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:01:40,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860912 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:01:45,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:01:45,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884048 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:01:50,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:01:50,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836612 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:01:55,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:01:55,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728912 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:00,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:00,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270299 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:05,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:05,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873575 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:10,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:10,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000666161 s, 6.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:15,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:15,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799201 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:20,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:20,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875232 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:25,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:25,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115504 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:31,043::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:02:31,045::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:02:31,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:31,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0167679 s, 122 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:31,222::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0167679 s, 122 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:02:31,226::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:02:31,227::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:02:31,232::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:02:31,239::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:02:31,243::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:02:31,245::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:02:31,247::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:02:31,250::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:02:31,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:31,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:31,785::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:02:31,790::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:02:31,803::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:02:31,806::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:02:31,808::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:02:31,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-13944::WARNING::2013-03-17 08:02:31,944::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:02:31,952::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:02:32,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:32,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:32,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00701287 s, 584 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:35,686::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:02:35,688::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:02:35,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:35,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0126546 s, 162 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:35,731::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0126546 s, 162 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:02:35,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:02:35,732::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:02:35,734::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:02:35,735::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:35,736::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:35,736::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:35,736::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:35,737::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:35,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:36,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:36,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:36,167::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:02:36,170::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:02:36,171::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:02:36,172::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:02:36,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:36,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:36,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:36,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189656 s, 216 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:02:41,955::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:02:41,957::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:42,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:42,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112631 s, 364 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:02:42,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:02:42,377::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:02:42,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:02:46,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:46,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124056 s, 330 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:02:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:02:52,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784188 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:02:56,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:02:56,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160606 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:03:02,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:03:02,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200993 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:03:06,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:03:06,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794768 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:03:12,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:03:12,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849246 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:03:16,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:03:16,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163776 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:03:22,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:03:22,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710837 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:03:26,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:03:26,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909731 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:03:32,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:03:32,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889118 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:03:36,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:03:36,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109216 s, 375 kB/s\n'; <rc> = 0 >Thread-13945::WARNING::2013-03-17 08:03:42,437::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:03:42,442::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:03:42,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:03:42,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013786 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:03:46,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:03:46,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875915 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:03:52,445::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:03:52,446::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:03:52,862::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:03:52,869::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:03:52,870::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:03:52,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:03:52,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855405 s, 479 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:03:56,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:03:56,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138678 s, 295 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:04:03,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:04:03,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815252 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:04:06,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:04:06,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793522 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:04:13,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:04:13,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733857 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:04:16,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:04:17,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146945 s, 279 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:04:23,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:04:23,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149732 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:04:27,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:04:27,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805611 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:04:33,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:04:33,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259677 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:04:37,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:04:37,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136601 s, 300 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:04:43,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:04:43,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149056 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:04:47,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:04:47,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137578 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13946::WARNING::2013-03-17 08:04:52,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:04:52,902::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:04:53,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:04:53,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969348 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:04:57,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:04:57,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116964 s, 350 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:05:02,908::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:05:02,909::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:05:02,910::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:05:03,353::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:05:03,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:05:03,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:05:03,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:05:03,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167324 s, 245 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:05:07,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:05:07,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149098 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:05:13,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:05:13,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785426 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:05:17,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:05:17,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200631 s, 204 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:05:23,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:05:23,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771967 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:05:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:05:27,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784141 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:05:33,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:05:33,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176549 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:05:37,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:05:37,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163744 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:05:43,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:05:43,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837212 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:05:47,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:05:47,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152704 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:05:53,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:05:53,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135767 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:05:57,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:05:57,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013621 s, 301 kB/s\n'; <rc> = 0 >Thread-13947::WARNING::2013-03-17 08:06:03,388::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:06:03,391::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:06:03,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:06:03,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138098 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:06:07,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:06:07,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804603 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:06:13,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:06:13,395::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:06:13,793::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:06:13,800::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:06:13,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:06:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:06:14,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010459 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:06:17,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:06:17,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156256 s, 262 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:06:24,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:06:24,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105823 s, 387 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:06:27,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:06:27,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141951 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:06:34,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:06:34,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841708 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:06:37,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:06:37,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135722 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:06:44,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:06:44,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946452 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:06:47,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:06:47,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167546 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:06:54,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:06:54,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151731 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:06:57,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:06:57,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00971585 s, 422 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:04,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:04,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147462 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:07,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:07,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105026 s, 3.9 MB/s\n'; <rc> = 0 >Thread-13948::WARNING::2013-03-17 08:07:13,826::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:07:13,830::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:07:14,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:14,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101629 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:17,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:17,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00463001 s, 885 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:07:23,832::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:07:23,834::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:07:24,252::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:07:24,259::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:07:24,260::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:24,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:24,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00961376 s, 426 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:27,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:27,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107638 s, 381 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:34,667::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:07:34,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:07:34,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:34,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012996 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:34,696::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012996 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:07:34,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:07:34,697::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:07:34,699::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:07:34,700::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:34,700::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:34,701::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:34,703::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:34,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:34,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:35,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:35,124::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:07:35,125::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:07:35,127::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:07:35,129::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:07:35,129::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:07:35,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:35,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:35,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:35,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133529 s, 307 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:37,997::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:07:37,998::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:07:37,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:38,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0146964 s, 139 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:38,048::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0146964 s, 139 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:07:38,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:07:38,049::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:07:38,051::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:07:38,052::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:07:38,053::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:07:38,053::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:07:38,054::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:07:38,054::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:07:38,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:38,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:38,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:07:38,528::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:07:38,531::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:07:38,532::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:07:38,532::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:07:38,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:38,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:38,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:38,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138726 s, 295 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:45,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:45,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894656 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:48,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:48,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00233514 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:07:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:07:55,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080872 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:07:58,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:07:59,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129247 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:08:05,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:08:05,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818084 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:08:09,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:08:09,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938485 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:08:15,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:08:15,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837504 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:08:19,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:08:19,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145148 s, 2.8 MB/s\n'; <rc> = 0 >Thread-13949::WARNING::2013-03-17 08:08:24,290::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:08:24,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:08:25,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:08:25,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122393 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:08:29,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:08:29,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823772 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:08:34,296::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:08:34,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:08:34,707::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:08:34,714::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:08:34,715::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:08:35,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:08:36,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00661599 s, 619 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:08:39,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:08:39,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165457 s, 248 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:08:46,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:08:46,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815901 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:08:49,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:08:49,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137907 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:08:56,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:08:56,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786525 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:08:59,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:08:59,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106043 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:09:06,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:09:06,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896892 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:09:09,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:09:09,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729711 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:09:16,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:09:16,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143218 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:09:19,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:09:19,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142072 s, 288 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:09:26,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:09:26,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742164 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:09:29,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:09:29,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164531 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13950::WARNING::2013-03-17 08:09:34,739::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:09:34,743::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:09:36,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:09:36,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836414 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:09:39,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:09:39,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144793 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:09:44,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:09:44,747::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:09:45,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:09:45,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:09:45,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:09:46,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:09:46,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113298 s, 362 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:09:49,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:09:49,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126258 s, 324 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:09:56,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:09:56,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969665 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:09:59,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:09:59,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145672 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:10:06,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:10:06,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913988 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:10:09,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:10:09,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822955 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:10:16,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:10:16,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145905 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:10:19,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:10:19,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936693 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:10:26,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:10:26,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857391 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:10:29,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:10:29,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143383 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:10:36,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:10:36,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845103 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:10:39,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:10:39,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163757 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13951::WARNING::2013-03-17 08:10:45,193::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:10:45,197::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:10:46,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:10:46,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799286 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:10:49,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:10:49,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121356 s, 338 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:10:55,199::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:10:55,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:10:55,201::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:10:55,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:10:55,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:10:55,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:10:57,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:10:57,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00872095 s, 470 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:10:59,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:10:59,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00660991 s, 620 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:11:07,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:11:07,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098883 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:11:09,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:11:09,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157356 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:11:17,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:11:17,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131065 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:11:19,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:11:19,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856787 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:11:27,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:11:27,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165811 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:11:29,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:11:29,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146053 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:11:37,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:11:37,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130152 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:11:39,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:11:40,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132396 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:11:47,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:11:47,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796851 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:11:50,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:11:50,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175422 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13952::WARNING::2013-03-17 08:11:55,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:11:55,666::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:11:57,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:11:57,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822701 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:00,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:00,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148332 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:12:05,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:12:05,669::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:12:06,180::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:12:06,189::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:12:06,191::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:07,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:07,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168027 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:10,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:10,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013508 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:17,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:17,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137727 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:20,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:20,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799553 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:27,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:27,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147125 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:30,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:30,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880219 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:37,803::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:12:37,805::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:12:37,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:37,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125095 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:37,841::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125095 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:12:37,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:12:37,841::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:12:37,843::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:12:37,844::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:37,845::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:37,845::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:37,845::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:37,846::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:37,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:38,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:38,287::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:12:38,288::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:12:38,292::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:12:38,293::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:12:38,294::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:12:38,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:38,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:38,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:38,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143158 s, 286 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:40,282::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:12:40,286::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:12:40,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:40,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0132552 s, 155 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:40,335::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0132552 s, 155 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:12:40,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:12:40,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:12:40,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:12:40,339::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:12:40,339::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:12:40,340::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:12:40,340::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:12:40,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:12:40,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:40,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:40,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:12:40,751::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:12:40,753::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:12:40,755::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:12:40,755::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:12:40,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:41,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:41,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:41,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160143 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:48,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:48,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760905 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:12:51,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:12:51,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084742 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:12:58,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:12:58,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818012 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:13:01,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:13:01,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173685 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13953::WARNING::2013-03-17 08:13:06,217::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:13:06,220::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:13:08,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:13:09,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102909 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:13:11,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:13:11,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917433 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:13:16,222::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:13:16,223::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:13:16,666::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:13:16,673::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:13:16,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:13:19,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:13:19,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223128 s, 184 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:13:21,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:13:21,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0285009 s, 144 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:13:29,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:13:29,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819201 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:13:31,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:13:31,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162447 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:13:39,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:13:39,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752439 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:13:41,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:13:41,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999442 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:13:49,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:13:49,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740011 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:13:51,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:13:51,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150395 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:13:59,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:13:59,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147601 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:14:01,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:14:01,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109025 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:14:09,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:14:09,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101586 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:14:11,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:14:11,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123171 s, 3.3 MB/s\n'; <rc> = 0 >Thread-13954::WARNING::2013-03-17 08:14:16,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:14:16,712::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:14:19,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:14:19,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155676 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:14:21,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:14:21,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149834 s, 273 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:14:26,720::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:14:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:14:27,288::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:14:27,297::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:14:27,299::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:14:29,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:14:29,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701024 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:14:31,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:14:31,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156618 s, 262 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:14:39,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:14:39,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302486 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:14:41,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:14:41,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150702 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:14:49,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:14:49,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807217 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:14:51,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:14:51,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186335 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:14:59,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:14:59,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159144 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:15:01,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:15:01,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008533 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:15:09,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:15:09,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801843 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:15:11,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:15:11,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748164 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:15:19,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:15:20,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263288 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:15:21,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:15:21,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991416 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13955::WARNING::2013-03-17 08:15:27,331::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:15:27,335::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:15:30,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:15:30,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810546 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:15:31,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:15:31,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766515 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:15:37,337::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:15:37,338::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:15:37,792::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:15:37,799::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:15:37,800::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:15:40,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:15:40,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887928 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:15:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:15:41,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147476 s, 278 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:15:50,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:15:50,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110584 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:15:51,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:15:52,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183679 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:16:00,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:16:00,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178006 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:16:02,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:16:02,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130027 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:16:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:16:10,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079724 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:16:12,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:16:12,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00735277 s, 557 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:16:20,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:16:20,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152424 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:16:22,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:16:22,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014689 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:16:30,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:16:30,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017551 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:16:32,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:16:32,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175208 s, 2.3 MB/s\n'; <rc> = 0 >Thread-13956::WARNING::2013-03-17 08:16:37,826::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:16:37,830::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:16:40,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:16:40,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101445 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:16:42,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:16:42,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157001 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:16:47,831::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:16:47,832::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:16:47,833::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:16:48,230::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:16:48,237::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:16:48,238::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:16:50,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:16:50,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145234 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:16:52,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:16:52,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014389 s, 285 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:00,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:00,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748595 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:02,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:02,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030462 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:10,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:10,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159766 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:12,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:12,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014475 s, 283 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:20,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:20,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894202 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:22,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:22,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128099 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:30,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:31,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206596 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:32,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:32,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101058 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:41,052::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:17:41,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:17:41,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:41,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00139372 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:41,095::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00139372 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:17:41,095::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:17:41,095::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:17:41,097::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:17:41,098::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:17:41,099::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:17:41,100::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:17:41,100::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:17:41,101::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:17:41,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:41,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:41,528::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:17:41,529::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:17:41,532::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:17:41,534::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:17:41,534::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:17:41,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:41,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:17:41,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:41,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00582575 s, 703 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:42,523::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:17:42,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:17:42,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:42,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00881823 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:42,569::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00881823 s, 232 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:17:42,569::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:17:42,570::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:17:42,572::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:17:42,574::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:17:42,575::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:17:42,575::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:17:42,576::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:17:42,576::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:17:42,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:43,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:43,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:17:43,008::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:17:43,011::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:17:43,012::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:17:43,012::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:17:43,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:43,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:43,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:43,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179384 s, 228 kB/s\n'; <rc> = 0 >Thread-13957::WARNING::2013-03-17 08:17:48,271::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:17:48,276::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:17:51,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:17:52,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840339 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:17:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:17:53,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195723 s, 209 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:17:58,278::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:17:58,279::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:17:58,699::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:17:58,706::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:17:58,707::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:18:02,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:18:02,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157124 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:18:03,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:18:03,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187746 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:18:12,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:18:12,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922196 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:18:13,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:18:13,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161976 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:18:22,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:18:22,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829952 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:18:23,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:18:23,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998779 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:18:32,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:18:32,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774342 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:18:33,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:18:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878659 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:18:42,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:18:42,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192568 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:18:43,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:18:43,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122797 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:18:52,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:18:52,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015203 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:18:53,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:18:53,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015796 s, 2.6 MB/s\n'; <rc> = 0 >Thread-13958::WARNING::2013-03-17 08:18:58,748::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:18:58,752::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:19:02,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:19:02,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119449 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:19:03,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:19:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759319 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:19:08,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:19:08,755::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:19:09,179::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:19:09,186::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:19:09,187::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:19:12,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:19:12,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117538 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:19:13,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:19:13,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142129 s, 288 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:19:22,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:19:22,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965893 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:19:23,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:19:23,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132991 s, 308 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:19:32,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:19:32,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837742 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:19:33,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:19:33,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088199 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:19:42,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:19:42,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001487 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:19:43,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:19:44,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112841 s, 363 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:19:52,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:19:52,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104193 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:19:54,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:19:54,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130705 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:20:02,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:20:02,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137907 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:20:04,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:20:04,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150872 s, 271 kB/s\n'; <rc> = 0 >Thread-13959::WARNING::2013-03-17 08:20:09,216::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:20:09,221::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:20:13,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:20:13,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811554 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:20:14,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:20:14,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146694 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:20:19,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:20:19,229::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:20:19,658::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:20:19,665::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:20:19,666::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:20:23,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:20:23,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919916 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:20:24,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:20:24,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203828 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:20:33,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:20:33,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915547 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:20:34,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:20:34,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722679 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:20:43,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:20:43,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102165 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:20:44,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:20:44,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172634 s, 237 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:20:53,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:20:53,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124167 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:20:54,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:20:54,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162057 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:21:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:21:03,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139251 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:21:04,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:21:04,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996665 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:21:13,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:21:13,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127069 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:21:14,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:21:14,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100945 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13960::WARNING::2013-03-17 08:21:19,698::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:21:19,701::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:21:23,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:21:23,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828246 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:21:24,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:21:24,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164025 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:21:29,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:21:29,712::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:21:30,129::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:21:30,136::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:21:30,138::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:21:33,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:21:33,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017775 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:21:34,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:21:34,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111813 s, 366 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:21:43,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:21:43,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783211 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:21:44,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:21:44,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997894 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:21:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:21:53,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170045 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:21:54,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:21:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818696 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:03,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:03,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743345 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:04,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:04,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0099018 s, 414 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:13,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:13,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138028 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:14,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:14,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152006 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:24,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:24,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00993445 s, 412 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:24,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:24,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295966 s, 1.4 MB/s\n'; <rc> = 0 >Thread-13961::WARNING::2013-03-17 08:22:30,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:22:30,168::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:22:34,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:34,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879959 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:34,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:34,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812941 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:22:40,169::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:22:40,170::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:22:40,171::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:22:40,626::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:22:40,633::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:22:40,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,165::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:22:44,167::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:22:44,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:44,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0102948 s, 199 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:44,208::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0102948 s, 199 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:22:44,209::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:22:44,209::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:22:44,211::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:22:44,212::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,213::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,213::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,214::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,214::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:44,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:44,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:44,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:22:44,651::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:22:44,652::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:22:44,653::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:22:44,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:44,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:22:44,985::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:22:44,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:45,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:45,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:45,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0122196 s, 168 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:45,136::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0122196 s, 168 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:22:45,146::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:22:45,147::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:22:45,152::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:22:45,157::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:22:45,161::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:22:45,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00732058 s, 560 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:45,163::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:22:45,172::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:22:45,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:22:45,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:45,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:45,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:22:45,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:22:45,651::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:22:45,653::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:22:45,653::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:22:45,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:46,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:46,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:46,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125818 s, 326 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:22:55,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:22:55,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800655 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:22:56,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:22:56,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109957 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:23:05,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:23:05,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715484 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:23:06,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:23:06,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161536 s, 254 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:23:15,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:23:15,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150454 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:23:16,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:23:16,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154854 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:23:25,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:23:25,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135177 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:23:26,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:23:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105872 s, 387 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:23:35,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:23:35,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800665 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:23:36,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:23:36,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140455 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13962::WARNING::2013-03-17 08:23:40,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:23:40,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:23:45,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:23:45,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811033 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:23:46,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:23:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141467 s, 290 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:23:50,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:23:50,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:23:51,110::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:23:51,117::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:23:51,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:23:55,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:23:55,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074846 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:23:56,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:23:56,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152346 s, 269 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:24:05,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:24:05,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156312 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:24:06,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:24:06,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181127 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:24:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:24:15,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807967 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:24:16,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:24:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759853 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:24:25,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:24:25,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143949 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:24:26,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:24:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017018 s, 241 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:24:36,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:24:36,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148163 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:24:36,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:24:36,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143594 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:24:46,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:24:46,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151151 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:24:46,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:24:46,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019392 s, 211 kB/s\n'; <rc> = 0 >Thread-13963::WARNING::2013-03-17 08:24:51,148::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:24:51,154::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:24:56,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:24:56,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00358043 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:24:56,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:24:56,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819026 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:25:01,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:25:01,158::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:25:01,681::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:25:01,687::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:25:01,688::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:25:06,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:25:06,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078474 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:25:06,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:25:06,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0570868 s, 71.8 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:25:16,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:25:16,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157343 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:25:16,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:25:16,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162759 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:25:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:25:26,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841218 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:25:26,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:25:26,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010245 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:25:36,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:25:36,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759037 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:25:36,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:25:36,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134077 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:25:46,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:25:46,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136215 s, 301 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:25:46,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:25:47,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00600315 s, 682 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:25:56,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:25:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146153 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:25:57,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:25:57,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917583 s, 4.5 MB/s\n'; <rc> = 0 >Thread-13964::WARNING::2013-03-17 08:26:01,724::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:26:01,727::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:26:06,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:26:06,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.049685 s, 82.4 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:26:07,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:26:07,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802915 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:26:11,730::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:26:11,731::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:26:12,207::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:26:12,213::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:26:12,215::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:26:16,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:26:16,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0099528 s, 412 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:26:17,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:26:17,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174148 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:26:26,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:26:26,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826899 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:26:27,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:26:27,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922698 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:26:37,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:26:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110926 s, 369 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:26:37,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:26:37,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150726 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:26:47,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:26:47,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155592 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:26:47,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:26:47,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866592 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:26:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:26:57,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084574 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:26:57,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:26:57,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823004 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:07,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:07,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:07,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021081 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:07,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110501 s, 3.7 MB/s\n'; <rc> = 0 >Thread-13965::WARNING::2013-03-17 08:27:12,247::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:27:12,256::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:27:17,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:17,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108344 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:17,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:17,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930565 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:27:22,268::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:27:22,270::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:27:22,663::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:27:22,670::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:27:22,671::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:27,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:27,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192058 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:27,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:27,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154227 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:37,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:37,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863522 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:37,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:37,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898943 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:47,529::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:27:47,531::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:27:47,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:47,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000906876 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:47,561::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000906876 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:27:47,561::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:27:47,561::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:27:47,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:27:47,564::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:47,565::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:47,565::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:47,566::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:47,566::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:47,666::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:27:47,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:27:47,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:47,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000991111 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:47,720::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000991111 s, 2.1 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:27:47,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:27:47,721::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:27:47,723::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:27:47,725::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-22::DEBUG::2013-03-17 08:27:48,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:48,017::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:27:48,024::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:27:48,024::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-20::DEBUG::2013-03-17 08:27:48,027::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:27:48,028::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:27:48,033::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:27:48,038::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::WARNING::2013-03-17 08:27:48,042::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:27:48,047::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:27:48,049::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:27:48,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:48,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:48,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:48,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:48,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:48,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:27:48,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:27:48,724::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:27:48,727::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:27:48,729::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:27:48,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:48,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167551 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:49,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:49,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112263 s, 365 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:27:58,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:27:58,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100349 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:27:59,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:27:59,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847827 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:28:08,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:28:08,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154434 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:28:09,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:28:09,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170816 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:28:18,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:28:18,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124794 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:28:19,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:28:19,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070651 s, 5.8 MB/s\n'; <rc> = 0 >Thread-13966::WARNING::2013-03-17 08:28:22,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:28:22,715::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:28:28,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:28:28,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164697 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:28:29,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:28:29,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019408 s, 2.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:28:32,717::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:28:32,718::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:28:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:28:33,198::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:28:33,205::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:28:33,206::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:28:38,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:28:39,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150507 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:28:39,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:28:39,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894523 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:28:49,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:28:49,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856982 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:28:49,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:28:49,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821029 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:28:59,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:28:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104642 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:28:59,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:28:59,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00239972 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:29:09,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:29:09,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872461 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:29:09,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:29:09,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868793 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:29:19,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:29:19,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088713 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:29:19,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:29:19,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140708 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:29:29,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:29:29,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163145 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:29:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:29:29,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766292 s, 5.3 MB/s\n'; <rc> = 0 >Thread-13967::WARNING::2013-03-17 08:29:33,235::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:29:33,240::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:29:39,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:29:39,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105956 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:29:39,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:29:39,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818627 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:29:43,242::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:29:43,244::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:29:43,665::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:29:43,671::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:29:43,672::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:29:49,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:29:49,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973092 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:29:50,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:29:50,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150891 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:29:59,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:29:59,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109643 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:30:00,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:30:00,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141458 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:30:09,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:30:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796763 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:30:10,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:30:10,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147827 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:30:19,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:30:19,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151677 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:30:20,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:30:20,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167533 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:30:29,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:30:29,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179612 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:30:30,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:30:30,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013976 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:30:39,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:30:39,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154642 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:30:40,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:30:40,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149483 s, 2.7 MB/s\n'; <rc> = 0 >Thread-13968::WARNING::2013-03-17 08:30:43,697::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:30:43,703::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:30:49,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:30:49,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165642 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:30:50,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:30:50,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799963 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:30:53,707::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:30:53,709::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:30:54,119::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:30:54,125::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:30:54,127::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:30:59,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:30:59,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007511 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:31:00,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:31:00,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150556 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:31:09,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:31:09,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775452 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:31:10,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:31:10,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946022 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:31:19,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:31:19,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806737 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:31:20,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:31:20,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090344 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:31:29,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:31:29,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124873 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:31:30,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:31:30,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825869 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:31:39,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:31:39,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916856 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:31:40,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:31:40,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873065 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:31:49,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:31:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138078 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:31:51,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:31:51,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779089 s, 5.3 MB/s\n'; <rc> = 0 >Thread-13969::WARNING::2013-03-17 08:31:54,171::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:31:54,176::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:31:59,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:31:59,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771751 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:01,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:01,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142428 s, 288 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:32:04,185::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:32:04,186::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:32:04,592::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:32:04,599::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:32:04,601::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:09,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:09,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121894 s, 336 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:11,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903859 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:19,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:19,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162148 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:21,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:21,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830968 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:29,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:29,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789866 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:31,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:31,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830045 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:39,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:39,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139919 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:41,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:41,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130618 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:49,978::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:32:49,984::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:32:49,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:50,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000876312 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:50,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000876312 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:32:50,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:32:50,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:32:50,020::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:32:50,022::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:50,023::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:50,023::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:50,023::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:50,024::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:50,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:50,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:50,460::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:32:50,461::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:32:50,464::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:32:50,465::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:32:50,465::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:32:50,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:50,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:32:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:32:50,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190479 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:51,460::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:32:51,462::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:32:51,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:51,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0117039 s, 175 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:51,514::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0117039 s, 175 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:32:51,514::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:32:51,514::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:32:51,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:32:51,527::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:32:51,528::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:32:51,528::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:32:51,529::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:32:51,529::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:32:51,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:51,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:51,981::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:32:51,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:32:51,984::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:32:51,986::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:32:51,986::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:32:51,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:52,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:32:52,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:32:52,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00410633 s, 997 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:33:00,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:33:00,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014104 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:33:02,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:33:02,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250877 s, 1.6 MB/s\n'; <rc> = 0 >Thread-13970::WARNING::2013-03-17 08:33:04,632::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:33:04,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:33:10,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:33:10,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814774 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:33:12,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:33:12,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131224 s, 3.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:33:14,638::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:33:14,639::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:33:15,073::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:33:15,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:33:15,081::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:33:20,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:33:20,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791733 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:33:22,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:33:22,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832657 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:33:30,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:33:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196995 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:33:32,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:33:32,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160884 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:33:41,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:33:41,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765173 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:33:42,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:33:42,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075776 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:33:51,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:33:51,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0035783 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:33:52,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:33:52,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015883 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:34:01,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:34:01,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016468 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:34:02,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:34:02,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170399 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:34:11,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:34:11,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136092 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:34:12,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:34:13,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083558 s, 4.9 MB/s\n'; <rc> = 0 >Thread-13971::WARNING::2013-03-17 08:34:15,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:34:15,128::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:34:21,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:34:21,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037109 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:34:23,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:34:23,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102981 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:34:25,130::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:34:25,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:34:25,133::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:34:25,558::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:34:25,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:34:25,566::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:34:31,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:34:31,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109843 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:34:33,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:34:33,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152439 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:34:41,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:34:41,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834143 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:34:43,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:34:43,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131627 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:34:51,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:34:51,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142902 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:34:53,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:34:53,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153312 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:35:01,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:35:01,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078846 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:35:03,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:35:03,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134997 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:35:11,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:35:11,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149829 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:35:13,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:35:13,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766366 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:35:21,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:35:21,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00897294 s, 456 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:35:23,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:35:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747025 s, 5.5 MB/s\n'; <rc> = 0 >Thread-13972::WARNING::2013-03-17 08:35:25,603::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:35:25,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:35:31,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:35:31,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788679 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:35:33,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:35:33,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837328 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:35:35,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:35:35,618::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:35:36,029::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:35:36,035::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:35:36,037::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:35:41,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:35:41,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850691 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:35:43,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:35:43,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133193 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:35:51,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:35:51,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165942 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:35:53,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:35:53,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148013 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:36:01,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:36:01,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783275 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:36:03,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:36:03,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133591 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:36:11,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:36:11,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184055 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:36:13,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:36:14,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142529 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:36:21,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:36:21,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125696 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:36:24,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:36:24,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103796 s, 395 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:36:31,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:36:31,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143414 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:36:34,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:36:34,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111539 s, 3.7 MB/s\n'; <rc> = 0 >Thread-13973::WARNING::2013-03-17 08:36:36,064::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:36:36,068::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:36:41,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:36:41,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173864 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:36:44,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:36:44,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137628 s, 298 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:36:46,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:36:46,071::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:36:46,513::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:36:46,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:36:46,522::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:36:51,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:36:51,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747891 s, 548 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:36:54,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:36:54,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849261 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:01,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:02,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00791091 s, 518 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:04,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:04,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188639 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:12,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:12,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011674 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:14,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:14,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720276 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:22,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:22,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103363 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:24,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:24,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848578 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:32,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:32,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100066 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:34,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:34,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759467 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:42,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:42,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013907 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:44,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:44,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213513 s, 1.9 MB/s\n'; <rc> = 0 >Thread-13974::WARNING::2013-03-17 08:37:46,548::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:37:46,552::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:37:52,288::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:37:52,290::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:37:52,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:52,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000898193 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:52,319::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000898193 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:37:52,320::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:37:52,320::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:37:52,322::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:37:52,323::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:37:52,324::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:37:52,324::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:37:52,324::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:37:52,325::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:37:52,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:52,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:52,766::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:37:52,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:37:52,769::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:37:52,770::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:37:52,771::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:37:52,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:53,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:37:53,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:37:53,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920051 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:54,810::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:37:54,812::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:37:54,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:54,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00986784 s, 208 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:54,864::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00986784 s, 208 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:37:54,864::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:37:54,865::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:37:54,866::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:37:54,867::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:37:54,868::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:37:54,869::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:37:54,869::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:37:54,869::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:37:54,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:55,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:55,311::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:37:55,312::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:37:55,315::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:37:55,316::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:37:55,317::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:37:55,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:37:55,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:37:55,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191512 s, 214 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:37:56,554::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:37:56,555::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:37:57,024::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:37:57,030::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:37:57,031::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:38:03,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:38:03,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017669 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:38:05,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:38:05,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0324424 s, 126 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:38:13,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:38:13,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129246 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:38:15,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:38:15,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145453 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:38:23,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:38:23,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150511 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:38:25,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:38:25,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078123 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:38:33,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:38:33,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156926 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:38:36,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:38:36,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158607 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:38:43,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:38:43,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790998 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:38:46,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:38:46,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992131 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:38:53,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:38:53,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138478 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:38:56,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:38:56,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023838 s, 172 kB/s\n'; <rc> = 0 >Thread-13975::WARNING::2013-03-17 08:38:57,062::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:38:57,067::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:39:03,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:39:03,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166919 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:39:06,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:39:06,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0256581 s, 160 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:39:07,071::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:39:07,072::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:39:07,504::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:39:07,510::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:39:07,512::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:39:13,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:39:13,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074156 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:39:16,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:39:16,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851489 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:39:23,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:39:23,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996706 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:39:26,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:39:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0047082 s, 870 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:39:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:39:33,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891202 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:39:36,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:39:36,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790085 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:39:43,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:39:43,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796826 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:39:46,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:39:46,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167871 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:39:53,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:39:53,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102232 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:39:56,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:39:56,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148037 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:40:03,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:40:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141881 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:40:07,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:40:07,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106783 s, 384 kB/s\n'; <rc> = 0 >Thread-13976::WARNING::2013-03-17 08:40:07,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:40:07,563::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:40:13,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:40:13,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854453 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:40:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:40:17,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.030745 s, 133 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:40:17,566::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:40:17,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:40:17,569::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:40:18,006::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:40:18,012::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:40:18,014::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:40:23,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:40:23,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145747 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:40:27,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:40:27,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831386 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:40:33,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:40:33,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947027 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:40:37,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:40:37,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907557 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:40:43,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:40:43,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00260639 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:40:47,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:40:47,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866297 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:40:53,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:40:53,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170479 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:40:57,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:40:57,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782656 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:41:03,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:41:03,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769452 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:41:07,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:07,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202159 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:41:13,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:41:14,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730131 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:41:17,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:17,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110907 s, 3.7 MB/s\n'; <rc> = 0 >Thread-13977::WARNING::2013-03-17 08:41:18,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:41:18,049::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 08:41:24,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:41:24,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122331 s, 335 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:41:28,066::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:41:28,067::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:28,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:28,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00714221 s, 573 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:41:28,507::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:41:28,513::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:41:28,514::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:41:34,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:41:34,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932605 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:41:38,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:38,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876505 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:41:44,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:41:44,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100242 s, 409 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:41:48,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:48,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943752 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:41:54,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:41:54,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134191 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:41:58,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:41:58,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761394 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:04,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:04,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144732 s, 283 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:08,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:08,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187694 s, 218 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:14,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175846 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:18,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:18,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174694 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:24,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:24,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973243 s, 4.2 MB/s\n'; <rc> = 0 >Thread-13978::WARNING::2013-03-17 08:42:28,540::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:42:28,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:42:28,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:28,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131319 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:34,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:34,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130429 s, 314 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:42:38,546::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:42:38,547::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:38,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:38,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148841 s, 275 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:42:39,001::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:42:39,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:42:39,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:44,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:44,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189533 s, 216 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:48,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:48,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0025652 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:54,511::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:42:54,512::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:42:54,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:54,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0148813 s, 138 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:54,554::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0148813 s, 138 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:42:54,555::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:42:54,555::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:42:54,557::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:42:54,559::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:54,560::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:54,560::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:54,561::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:54,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:54,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:54,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:54,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:42:54,991::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:42:54,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:42:54,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:42:54,995::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:42:54,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:55,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:42:55,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:42:55,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112359 s, 365 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:58,907::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:42:58,909::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:42:58,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:58,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0102264 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:58,953::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0102264 s, 200 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:42:58,953::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:42:58,954::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:42:58,955::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:42:58,956::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:42:58,957::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:42:58,958::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:42:58,958::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:42:58,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:42:58,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:59,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:59,393::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:42:59,394::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:42:59,396::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:42:59,397::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:42:59,398::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:42:59,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:59,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:42:59,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:42:59,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00966687 s, 424 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:43:05,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:43:05,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100667 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:43:09,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:43:09,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988227 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:43:15,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:43:15,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841025 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:43:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:43:19,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141031 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:43:25,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:43:25,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162009 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:43:29,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:43:30,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073631 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:43:35,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:43:35,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013569 s, 3.0 MB/s\n'; <rc> = 0 >Thread-13979::WARNING::2013-03-17 08:43:39,034::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:43:39,038::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:43:40,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:43:40,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089527 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:43:45,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:43:45,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793436 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:43:49,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:43:49,041::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:43:49,453::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:43:49,460::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:43:49,461::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:43:50,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:43:50,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00950779 s, 431 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:43:55,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:43:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163121 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:44:00,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:44:00,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924399 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:44:05,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:44:05,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152772 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:44:10,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:44:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155471 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:44:15,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:44:15,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077064 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:44:20,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:44:20,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809513 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:44:25,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:44:25,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850011 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:44:30,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:44:30,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861441 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:44:35,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:44:35,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00347966 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:44:40,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:44:40,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825148 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:44:45,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:44:45,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961522 s, 4.3 MB/s\n'; <rc> = 0 >Thread-13980::WARNING::2013-03-17 08:44:49,507::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:44:49,510::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:44:50,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:44:50,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831598 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:44:55,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:44:55,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085256 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:44:59,512::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:44:59,514::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:44:59,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:44:59,996::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:44:59,997::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:45:00,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:45:00,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137875 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:45:05,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:45:06,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777949 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:45:10,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:45:10,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190535 s, 215 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:45:16,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:45:16,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125834 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:45:20,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:45:20,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125151 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:45:26,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:45:26,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176892 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:45:30,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:45:30,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130764 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:45:36,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:45:36,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787043 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:45:41,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:45:41,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809771 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:45:46,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:45:46,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00516569 s, 793 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:45:51,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:45:51,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910335 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:45:56,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:45:56,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167379 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13981::WARNING::2013-03-17 08:46:00,023::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:46:00,025::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:46:01,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:46:01,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719901 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:46:06,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:46:06,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904559 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:46:10,027::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:46:10,028::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:46:10,029::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:46:10,467::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:46:10,473::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:46:10,475::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:46:11,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:46:11,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107772 s, 380 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:46:16,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:46:16,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808057 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:46:21,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:46:21,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956477 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:46:26,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:46:26,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141891 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:46:31,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:46:31,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168443 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:46:36,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:46:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813165 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:46:41,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:46:41,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759316 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:46:46,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:46:46,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168044 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:46:51,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:46:51,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154473 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:46:56,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:46:56,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014291 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:47:01,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:47:01,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176586 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:06,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:06,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00887298 s, 462 kB/s\n'; <rc> = 0 >Thread-13982::WARNING::2013-03-17 08:47:10,505::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:47:10,507::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:47:11,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:47:11,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172137 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:16,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:16,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855297 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:47:20,513::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:47:20,514::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:47:20,914::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:47:20,920::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:47:20,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:47:21,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:47:21,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125919 s, 325 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:26,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:26,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116019 s, 353 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:47:31,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:47:31,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841093 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:36,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:36,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920421 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:47:41,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:47:41,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154258 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:46,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:46,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809305 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:47:51,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:47:52,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149815 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:56,793::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:47:56,794::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:47:56,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:56,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121111 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:56,827::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121111 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:47:56,828::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:47:56,828::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:47:56,830::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:47:56,832::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:47:56,832::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:47:56,833::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:47:56,833::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:47:56,834::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:47:56,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:57,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:57,288::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:47:57,289::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:47:57,292::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:47:57,293::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:47:57,293::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:47:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:57,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:47:57,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:47:57,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129232 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:02,068::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:48:02,069::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:48:02,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:02,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0175102 s, 117 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:02,132::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0175102 s, 117 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:48:02,135::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:48:02,135::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:48:02,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:48:02,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:02,139::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:02,139::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:02,139::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:02,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:02,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:02,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:02,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:02,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:48:02,599::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:48:02,601::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:48:02,601::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:48:02,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:02,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:02,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:03,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743099 s, 551 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:48:07,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:48:07,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0282677 s, 145 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:13,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:13,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881217 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:48:17,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:48:17,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774815 s, 5.3 MB/s\n'; <rc> = 0 >Thread-13983::WARNING::2013-03-17 08:48:20,959::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:48:20,962::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:48:23,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:23,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729647 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:48:27,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:48:27,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874695 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:48:30,964::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:48:30,965::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:48:31,385::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:48:31,392::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:48:31,393::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:48:33,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:33,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149764 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:48:37,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:48:37,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141674 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:43,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:43,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842223 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:48:47,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:48:48,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072193 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:48:53,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:48:53,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836222 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:48:58,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:48:58,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799209 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:49:03,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:49:03,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154574 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:49:08,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:49:08,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786627 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:49:13,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:49:13,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086873 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:49:18,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:49:18,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191672 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:49:23,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:49:23,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138425 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:49:28,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:49:28,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225419 s, 182 kB/s\n'; <rc> = 0 >Thread-13984::WARNING::2013-03-17 08:49:31,427::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:49:31,431::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:49:33,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:49:33,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763954 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:49:38,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:49:38,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818553 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:49:41,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:49:41,434::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:49:41,861::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:49:41,868::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:49:41,869::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:49:43,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:49:43,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100228 s, 409 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:49:48,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:49:48,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778557 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:49:53,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:49:53,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281004 s, 146 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:49:58,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:49:58,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967224 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:50:03,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:50:03,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693081 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:50:08,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:50:08,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129427 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:50:13,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:50:14,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163663 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:50:18,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:50:18,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172092 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:50:24,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:50:24,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00938442 s, 436 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:50:28,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:50:28,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146264 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:50:34,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:50:34,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0528999 s, 77.4 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:50:38,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:50:38,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990042 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13985::WARNING::2013-03-17 08:50:41,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:50:41,903::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:50:44,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:50:44,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829487 s, 494 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:50:48,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:50:48,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809512 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:50:51,907::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:50:51,908::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:50:52,364::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:50:52,371::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:50:52,373::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:50:54,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:50:54,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139123 s, 294 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:50:58,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:50:58,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110715 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:51:04,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:51:04,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131027 s, 313 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:51:08,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:51:08,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109529 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:51:14,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:51:14,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827534 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:51:18,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:51:18,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989195 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:51:24,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:51:24,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162688 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:51:28,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:51:28,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117242 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:51:34,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:51:34,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142671 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:51:38,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:51:38,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825732 s, 496 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:51:44,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:51:44,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141795 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:51:48,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:51:49,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727711 s, 5.6 MB/s\n'; <rc> = 0 >Thread-13986::WARNING::2013-03-17 08:51:52,399::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:51:52,403::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:51:54,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:51:54,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872207 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:51:59,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:51:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718275 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:52:02,405::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:52:02,406::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:52:02,407::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:52:02,821::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:52:02,828::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:52:02,829::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:52:04,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:52:04,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198974 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:09,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:09,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257538 s, 159 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:52:15,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:52:15,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815942 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:19,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:19,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144057 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:52:25,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:52:25,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858852 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:29,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:29,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00874334 s, 468 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:52:35,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:52:35,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880789 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:39,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:39,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136081 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:52:45,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:52:45,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135213 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:49,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:49,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00954731 s, 429 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:52:55,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:52:55,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149616 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:59,486::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:52:59,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:52:59,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:59,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135008 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:59,526::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135008 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:52:59,527::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:52:59,527::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:52:59,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:52:59,534::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:52:59,535::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:52:59,535::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:52:59,535::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:52:59,536::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:52:59,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:52:59,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:52:59,973::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:52:59,974::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:52:59,977::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:52:59,978::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:52:59,979::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:52:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:00,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:53:00,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:00,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00501897 s, 816 kB/s\n'; <rc> = 0 >Thread-13987::WARNING::2013-03-17 08:53:02,859::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:53:02,867::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:53:05,396::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:53:05,397::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:53:05,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:05,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00151104 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:05,432::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00151104 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:53:05,432::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:53:05,433::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:53:05,434::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:53:05,435::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:05,436::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:05,436::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:05,437::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:05,437::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:05,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:05,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:05,841::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:05,842::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:53:05,844::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:53:05,846::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:53:05,846::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:53:05,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:06,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:06,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:06,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00998671 s, 410 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:53:10,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:10,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133647 s, 306 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:53:12,870::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:53:12,873::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:53:13,290::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:53:13,297::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:53:13,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:53:16,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:16,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189939 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:53:20,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:20,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757838 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:26,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:26,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155356 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:53:30,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:30,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00251963 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:36,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:36,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858357 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:53:40,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:40,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864759 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:46,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:46,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953968 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:53:50,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:53:50,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141395 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:53:56,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:53:56,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163439 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:54:00,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:54:00,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143597 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:54:06,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:54:06,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159769 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:54:10,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:54:10,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142972 s, 2.9 MB/s\n'; <rc> = 0 >Thread-13988::WARNING::2013-03-17 08:54:13,347::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:54:13,350::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:54:16,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:54:16,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729103 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:54:20,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:54:20,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822435 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:54:23,352::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:54:23,353::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:54:23,788::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:54:23,795::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:54:23,796::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:54:26,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:54:26,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147162 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:54:30,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:54:30,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697558 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:54:36,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:54:36,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896768 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:54:40,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:54:40,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141001 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:54:46,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:54:47,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816566 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:54:50,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:54:50,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087719 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:54:57,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:54:57,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871928 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:55:00,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:55:00,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168524 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:55:07,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:55:07,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177611 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:55:10,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:55:10,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140232 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:55:17,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:55:17,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872619 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:55:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:55:21,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173321 s, 2.4 MB/s\n'; <rc> = 0 >Thread-13989::WARNING::2013-03-17 08:55:23,823::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:55:23,827::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:55:27,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:55:27,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920591 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:55:31,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:55:31,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881782 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:55:33,833::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:55:33,834::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:55:34,291::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:55:34,298::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:55:34,300::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:55:37,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:55:37,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149216 s, 275 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:55:41,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:55:41,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148729 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:55:47,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:55:47,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132266 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:55:51,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:55:51,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167975 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:55:57,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:55:57,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105068 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:56:01,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:56:01,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146686 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:56:07,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:56:07,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151187 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:56:11,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:56:11,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820522 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:56:17,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:56:17,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010171 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:56:21,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:56:21,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152185 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:56:27,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:56:27,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322818 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:56:31,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:56:31,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875468 s, 4.7 MB/s\n'; <rc> = 0 >Thread-13990::WARNING::2013-03-17 08:56:34,329::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:56:34,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:56:37,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:56:37,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173685 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:56:41,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:56:41,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974776 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:56:44,351::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:56:44,353::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:56:44,763::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:56:44,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:56:44,774::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:56:47,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:56:48,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100497 s, 408 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:56:51,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:56:51,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749127 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:56:58,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:56:58,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829626 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:57:01,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:57:01,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809287 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:57:08,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:57:08,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775635 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:57:11,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:57:11,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118924 s, 344 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:57:18,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:57:18,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148356 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:57:21,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:57:21,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188222 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:57:28,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:57:28,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777448 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:57:31,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:57:31,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809137 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:57:38,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:57:38,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162792 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:57:41,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:57:41,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977949 s, 4.2 MB/s\n'; <rc> = 0 >Thread-13991::WARNING::2013-03-17 08:57:44,803::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:57:44,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:57:48,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:57:48,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870014 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:57:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:57:51,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913703 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:57:54,817::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 08:57:54,818::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:57:54,819::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:57:55,264::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:57:55,270::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:57:55,272::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:57:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:57:58,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119381 s, 343 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:01,693::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 08:58:01,695::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:58:01,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:01,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0111973 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:01,754::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0111973 s, 183 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 08:58:01,754::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 08:58:01,755::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 08:58:01,757::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 08:58:01,758::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:58:01,758::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:58:01,759::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:58:01,759::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:58:01,760::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 08:58:01,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:02,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:02,231::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 08:58:02,232::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 08:58:02,236::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 08:58:02,237::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 08:58:02,238::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 08:58:02,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:02,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:02,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:02,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178836 s, 229 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:08,590::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 08:58:08,591::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:58:08,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:08,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00080476 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:08,618::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00080476 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 08:58:08,618::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 08:58:08,618::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 08:58:08,620::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 08:58:08,621::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:58:08,622::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:58:08,622::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:58:08,623::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:58:08,623::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 08:58:08,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:09,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:09,066::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:58:09,067::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 08:58:09,070::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 08:58:09,071::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 08:58:09,072::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 08:58:09,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:09,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:09,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:09,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130133 s, 315 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:12,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:12,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010046 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:19,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:19,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820417 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:22,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:22,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744175 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:29,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:29,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107346 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:32,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:32,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162327 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:39,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:39,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146171 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:42,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:42,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804457 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:58:49,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:49,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145123 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:58:52,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:58:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101102 s, 4.1 MB/s\n'; <rc> = 0 >Thread-13992::WARNING::2013-03-17 08:58:55,298::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 08:58:55,302::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 08:58:59,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:58:59,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826814 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:59:02,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:59:02,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876481 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 08:59:05,306::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 08:59:05,307::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 08:59:05,693::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 08:59:05,700::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 08:59:05,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 08:59:09,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:59:09,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204143 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:59:12,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:59:12,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085888 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:59:20,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:59:20,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795599 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:59:23,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:59:23,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759612 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:59:30,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:59:30,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137155 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:59:33,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:59:33,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855921 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:59:40,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:59:40,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159072 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:59:43,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:59:43,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828914 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 08:59:50,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 08:59:50,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168411 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 08:59:53,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 08:59:53,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155139 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:00:00,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:00:00,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814654 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:00:03,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:00:03,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116946 s, 3.5 MB/s\n'; <rc> = 0 >Thread-13993::WARNING::2013-03-17 09:00:05,727::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:00:05,731::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:00:10,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:00:10,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114739 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:00:13,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:00:13,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145457 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:00:15,772::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:00:15,774::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:00:16,268::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:00:16,274::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:00:16,276::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:00:20,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:00:20,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127952 s, 320 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:00:23,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:00:23,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731526 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:00:30,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:00:30,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163033 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:00:33,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:00:33,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152285 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:00:40,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:00:40,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889714 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:00:43,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:00:43,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150593 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:00:50,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:00:50,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176707 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:00:53,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:00:53,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074685 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:01:00,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:01:00,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161634 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:01:03,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:01:03,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00291879 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:01:10,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:01:10,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821725 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:01:13,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:01:13,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107842 s, 3.8 MB/s\n'; <rc> = 0 >Thread-13994::WARNING::2013-03-17 09:01:16,306::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:01:16,316::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:01:21,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:01:21,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151957 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:01:23,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:01:23,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000599416 s, 6.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:01:26,333::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:01:26,335::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:01:26,816::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:01:26,823::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:01:26,824::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:01:31,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:01:31,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137985 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:01:33,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:01:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933496 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:01:41,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:01:41,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824141 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:01:43,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:01:43,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788048 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:01:51,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:01:51,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138865 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:01:53,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:01:53,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173722 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:02:01,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:02:01,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157091 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:02:03,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:02:03,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137979 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:02:11,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:02:11,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159446 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:02:13,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:02:13,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.007993 s, 512 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:02:21,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:02:21,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966725 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:02:23,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:02:23,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0078611 s, 521 kB/s\n'; <rc> = 0 >Thread-13995::WARNING::2013-03-17 09:02:26,850::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:02:26,854::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:02:31,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:02:31,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218148 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:02:33,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:02:33,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162905 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:02:36,856::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:02:36,858::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:02:37,279::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:02:37,286::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:02:37,287::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:02:41,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:02:41,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00874414 s, 468 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:02:43,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:02:43,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181626 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:02:51,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:02:51,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142218 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:02:53,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:02:54,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00795542 s, 515 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:01,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:01,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128478 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:04,027::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:03:04,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:03:04,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:04,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0108884 s, 188 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:04,092::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0108884 s, 188 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:03:04,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:03:04,092::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:03:04,095::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:03:04,096::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:03:04,097::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:03:04,097::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:03:04,098::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:03:04,098::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:03:04,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:04,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:04,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:03:04,510::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:03:04,512::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:03:04,513::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:03:04,514::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:03:04,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:04,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:04,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:04,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100942 s, 406 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:11,890::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:03:11,891::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:03:11,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:11,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000763408 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:11,924::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000763408 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:03:11,924::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:03:11,925::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:03:11,926::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:03:11,927::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:11,928::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:11,928::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:11,929::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:11,929::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:11,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:12,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:12,357::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:12,358::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:03:12,360::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:03:12,361::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:03:12,362::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:03:12,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:12,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:12,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:12,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108453 s, 378 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:14,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:15,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838806 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:22,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:22,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939355 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:25,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:25,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014903 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:03:32,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:33,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158231 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:35,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:35,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114303 s, 3.6 MB/s\n'; <rc> = 0 >Thread-13996::WARNING::2013-03-17 09:03:37,315::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:03:37,321::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:03:43,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:43,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807966 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:45,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:45,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984385 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:03:47,324::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:03:47,326::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:03:47,327::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:03:47,810::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:03:47,817::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:03:47,818::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:03:53,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:03:53,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126371 s, 324 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:03:55,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:03:55,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860435 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:04:03,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:04:03,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835633 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:04:05,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:04:05,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136087 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:04:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:04:13,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228304 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:04:15,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:04:15,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0293674 s, 139 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:04:23,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:04:23,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167746 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:04:25,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:04:25,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804979 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:04:33,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:04:33,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166679 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:04:35,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:04:35,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013598 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:04:43,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:04:43,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104804 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:04:45,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:04:45,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164542 s, 2.5 MB/s\n'; <rc> = 0 >Thread-13997::WARNING::2013-03-17 09:04:47,842::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:04:47,846::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:04:53,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:04:53,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148826 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:04:55,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:04:55,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182612 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:04:57,849::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:04:57,850::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:04:58,256::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:04:58,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:04:58,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:05:03,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:05:03,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752221 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:05:05,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:05:05,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864714 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:05:13,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:05:13,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785637 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:05:15,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:05:15,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862235 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:05:23,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:05:23,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307622 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:05:25,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:05:25,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284638 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:05:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:05:33,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769039 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:05:35,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:05:35,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102376 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:05:44,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:05:44,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00953501 s, 430 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:05:45,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:05:45,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0033431 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:05:54,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:05:54,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139169 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:05:55,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:05:55,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741485 s, 5.5 MB/s\n'; <rc> = 0 >Thread-13998::WARNING::2013-03-17 09:05:58,300::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:05:58,306::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:06:04,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:06:04,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829651 s, 494 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:06:05,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:06:05,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103015 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:06:08,309::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:06:08,310::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:06:08,714::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:06:08,721::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:06:08,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:06:14,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:06:14,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178495 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:06:15,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:06:15,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154693 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:06:24,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:06:24,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010761 s, 381 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:06:25,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:06:25,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146711 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:06:34,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:06:34,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113136 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:06:35,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:06:35,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176981 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:06:44,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:06:44,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122423 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:06:45,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:06:45,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770861 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:06:54,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:06:54,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160612 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:06:55,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:06:56,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140638 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:07:04,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:07:04,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163481 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:07:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:07:06,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859052 s, 4.8 MB/s\n'; <rc> = 0 >Thread-13999::WARNING::2013-03-17 09:07:08,748::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:07:08,751::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:07:14,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:07:14,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085298 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:07:16,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:07:16,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155932 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:07:18,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:07:18,755::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:07:19,148::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:07:19,155::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:07:19,156::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:07:24,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:07:24,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788039 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:07:26,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:07:26,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144657 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:07:34,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:07:34,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148847 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:07:36,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:07:36,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809192 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:07:44,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:07:44,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140167 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:07:46,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:07:46,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998696 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:07:55,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:07:55,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129634 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:07:56,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:07:56,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154915 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:05,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:05,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134682 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:06,283::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:08:06,285::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:08:06,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:06,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140508 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:06,326::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140508 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:08:06,327::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:08:06,327::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:08:06,329::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:08:06,330::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:08:06,330::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:08:06,331::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:08:06,331::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:08:06,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:08:06,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:06,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:06,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:08:06,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:08:06,780::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:08:06,781::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:08:06,781::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:08:06,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:07,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:07,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:07,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00709426 s, 577 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:15,193::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:08:15,195::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:08:15,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:15,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0157912 s, 130 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:15,256::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0157912 s, 130 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:08:15,256::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:08:15,256::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:08:15,258::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:08:15,259::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:15,260::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:15,260::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:15,260::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:15,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:15,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:15,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:15,708::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:15,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:08:15,716::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:08:15,717::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:08:15,717::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:08:15,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:16,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:16,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:16,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105071 s, 390 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:17,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:17,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813651 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14000::WARNING::2013-03-17 09:08:19,186::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:08:19,192::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:08:26,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:26,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821903 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:27,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:27,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798648 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:08:29,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:08:29,196::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:08:29,628::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:08:29,635::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:08:29,636::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:08:36,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:36,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086404 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:37,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:37,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144999 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:46,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:46,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807678 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:47,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:47,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839379 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:08:56,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:08:56,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154886 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:08:57,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:08:57,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858504 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:09:06,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:09:06,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015674 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:09:07,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:09:07,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00317898 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:09:16,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:09:16,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131265 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:09:17,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:09:17,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769913 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:09:26,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:09:26,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141602 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:09:27,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:09:27,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143113 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14001::WARNING::2013-03-17 09:09:29,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:09:29,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:09:36,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:09:36,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788004 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:09:37,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:09:37,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104107 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:09:39,678::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:09:39,680::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:09:39,681::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:09:40,071::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:09:40,077::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:09:40,079::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:09:46,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:09:46,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722067 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:09:47,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:09:47,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150683 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:09:56,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:09:56,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731828 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:09:57,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:09:57,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919638 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:10:07,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:10:07,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569788 s, 719 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:10:07,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:10:07,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727304 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:10:17,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:10:17,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748682 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:10:17,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:10:17,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144375 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:10:27,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:10:27,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149728 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:10:27,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:10:27,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150439 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:10:37,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:10:37,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697807 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:10:37,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:10:37,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117317 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14002::WARNING::2013-03-17 09:10:40,106::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:10:40,110::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:10:47,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:10:47,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806077 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:10:47,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:10:47,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817582 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:10:50,112::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:10:50,113::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:10:50,564::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:10:50,570::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:10:50,572::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:10:57,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:10:57,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014166 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:10:57,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:10:57,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795618 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:11:07,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:11:07,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123693 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:11:07,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:11:08,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902265 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:11:17,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:11:17,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115084 s, 356 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:11:18,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:11:18,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076102 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:11:27,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:11:27,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155753 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:11:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:11:28,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979171 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:11:37,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:11:37,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904857 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:11:38,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:11:38,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144133 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:11:47,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:11:47,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0036645 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:11:48,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:11:48,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000676268 s, 6.1 MB/s\n'; <rc> = 0 >Thread-14003::WARNING::2013-03-17 09:11:50,602::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:11:50,608::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:11:57,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:11:57,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010028 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:11:58,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:11:58,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828884 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:12:00,611::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:12:00,612::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:12:01,037::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:12:01,044::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:12:01,046::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:12:07,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:12:07,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990317 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:12:08,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:12:08,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792711 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:12:18,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:12:18,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214216 s, 191 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:12:18,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:12:18,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111527 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:12:28,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:12:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088898 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:12:28,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:12:28,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860263 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:12:38,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:12:38,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853479 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:12:38,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:12:38,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00347845 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:12:48,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:12:48,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155285 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:12:48,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:12:48,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813149 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:12:58,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:12:58,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101886 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:12:58,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:12:58,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182205 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14004::WARNING::2013-03-17 09:13:01,074::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:13:01,077::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:13:08,523::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:13:08,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:13:08,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:08,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:08,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104049 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:08,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0022396 s, 914 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:08,638::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0022396 s, 914 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:13:08,641::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:13:08,643::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:13:08,647::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:13:08,659::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:13:08,662::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:13:08,664::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:13:08,667::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:13:08,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:13:08,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:09,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:09,114::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:13:09,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:13:09,117::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:13:09,118::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:13:09,119::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:13:09,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:09,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:09,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:09,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00396603 s, 1.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:13:11,080::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:13:11,081::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:13:11,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:13:11,516::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:13:11,517::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:18,750::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:13:18,753::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:13:18,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:18,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000743728 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:18,801::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000743728 s, 2.8 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:13:18,801::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:13:18,802::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:13:18,803::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:13:18,804::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:18,805::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:18,806::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:18,806::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:18,806::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:18,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:19,253::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:13:19,254::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:13:19,256::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:13:19,258::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:13:19,258::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:13:19,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:19,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:19,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00341445 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:19,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:19,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:19,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118449 s, 346 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:29,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831994 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:29,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:29,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855669 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:39,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143763 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:39,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:39,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086882 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:49,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:49,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132875 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:13:49,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:13:49,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108725 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:13:59,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:13:59,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153843 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:14:00,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:14:00,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782018 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:14:09,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:14:09,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103301 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:14:10,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:14:10,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00796252 s, 514 kB/s\n'; <rc> = 0 >Thread-14005::WARNING::2013-03-17 09:14:11,544::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:14:11,547::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:14:19,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:14:19,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803706 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:14:20,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:14:20,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841611 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:14:21,549::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:14:21,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:14:21,992::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:14:21,999::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:14:22,001::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:14:29,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:14:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910024 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:14:30,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:14:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776687 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:14:39,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:14:39,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905727 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:14:40,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:14:40,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880087 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:14:49,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:14:49,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884441 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:14:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:14:50,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814626 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:14:59,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:14:59,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165044 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:15:00,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:15:00,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010187 s, 402 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:15:10,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:15:10,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138697 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:15:11,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:15:11,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159358 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:15:20,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:15:20,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153625 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:15:21,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:15:21,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122533 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14006::WARNING::2013-03-17 09:15:22,030::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:15:22,036::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:15:30,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:15:30,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814451 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:15:31,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:15:31,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817273 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:15:32,038::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:15:32,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:15:32,041::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:15:32,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:15:32,508::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:15:32,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:15:40,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:15:40,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010808 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:15:41,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:15:41,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834141 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:15:50,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:15:50,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866491 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:15:51,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:15:51,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844033 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:16:00,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:16:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086238 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:16:01,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:16:01,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232386 s, 176 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:16:10,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:16:10,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139839 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:16:11,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:16:11,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00071491 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:16:20,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:16:20,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000647325 s, 6.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:16:21,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:16:21,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144873 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:16:30,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:16:30,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733824 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:16:32,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:16:32,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143848 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14007::WARNING::2013-03-17 09:16:32,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:16:32,545::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:16:40,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:16:40,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989333 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:16:42,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:16:42,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846997 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:16:42,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:16:42,550::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:16:42,997::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:16:43,004::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:16:43,005::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:16:50,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:16:50,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930025 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:16:52,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:16:52,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788726 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:17:00,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:17:00,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762391 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:17:02,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:02,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836451 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:17:10,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:17:10,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165948 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:17:12,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:12,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851342 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:17:20,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:17:20,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168953 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:17:22,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:22,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165381 s, 248 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:17:30,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:17:30,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093862 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:17:32,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:32,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159961 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:17:40,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:17:40,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140463 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:17:42,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:42,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316073 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14008::WARNING::2013-03-17 09:17:43,032::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:17:43,042::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:17:50,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:17:50,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078664 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:17:53,045::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:17:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:53,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:17:53,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117065 s, 350 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:17:53,532::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:17:53,537::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:17:53,538::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:00,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:00,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822591 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:03,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:03,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819505 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:10,745::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:18:10,747::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:18:10,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:10,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00122717 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:10,785::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00122717 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:18:10,786::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:18:10,786::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:18:10,788::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:18:10,789::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:10,789::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:10,790::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:10,790::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:10,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:10,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:11,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:11,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:18:11,234::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:18:11,236::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:18:11,238::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:18:11,238::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:18:11,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:11,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:11,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:11,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015995 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:13,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:13,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168904 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:21,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:21,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132817 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:23,391::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:18:23,393::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:18:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:23,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136367 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:23,444::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136367 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:18:23,445::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:18:23,445::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:18:23,447::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:18:23,448::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:18:23,448::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:18:23,449::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:18:23,449::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:18:23,449::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:18:23,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:23,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:23,870::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:18:23,871::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:18:23,874::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:18:23,875::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:18:23,876::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:18:23,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:24,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:24,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:24,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01307 s, 313 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:31,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:31,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146286 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:34,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:34,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841187 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:41,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856656 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:18:44,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:44,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112145 s, 365 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:18:51,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:18:51,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111936 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14009::WARNING::2013-03-17 09:18:53,566::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:18:53,570::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:18:54,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:18:54,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100125 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:19:01,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:19:01,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818128 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:19:03,573::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:19:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:19:04,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:19:04,020::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:19:04,021::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:19:04,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:19:04,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156621 s, 262 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:19:11,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:19:11,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808592 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:19:14,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:19:14,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164709 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:19:21,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:19:21,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149591 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:19:24,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:19:24,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108346 s, 378 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:19:31,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:19:32,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149046 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:19:34,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:19:34,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100681 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:19:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:19:42,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168643 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:19:44,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:19:44,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163757 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:19:52,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:19:52,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774026 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:19:55,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:19:55,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860654 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:20:02,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:20:02,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130608 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14010::WARNING::2013-03-17 09:20:04,064::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:20:04,066::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:20:05,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:20:05,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885327 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:20:12,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:20:12,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738609 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:20:14,071::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:20:14,073::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:20:14,520::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:20:14,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:20:14,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:20:15,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:20:15,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110299 s, 371 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:20:22,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:20:22,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155156 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:20:25,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:20:25,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143966 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:20:32,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:20:32,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785255 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:20:35,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:20:35,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754799 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:20:42,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:20:42,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934206 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:20:45,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:20:45,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158556 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:20:52,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:20:52,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143725 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:20:55,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:20:55,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122475 s, 334 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:21:02,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:21:02,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143084 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:21:05,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:21:05,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215791 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:21:12,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:21:12,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126968 s, 3.2 MB/s\n'; <rc> = 0 >Thread-14011::WARNING::2013-03-17 09:21:14,557::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:21:14,561::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:21:15,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:21:15,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021109 s, 194 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:21:22,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:21:22,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768772 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:21:24,562::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:21:24,563::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:21:24,564::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:21:25,018::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:21:25,024::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:21:25,026::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:21:25,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:21:25,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00800796 s, 511 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:21:32,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:21:32,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140124 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:21:35,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:21:35,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00957305 s, 428 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:21:42,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:21:42,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145836 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:21:45,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:21:45,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861407 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:21:52,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:21:52,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145876 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:21:56,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:21:56,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00519651 s, 788 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:22:02,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:22:02,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134868 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:22:06,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:22:06,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701988 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:22:12,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:22:12,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132715 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:22:16,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:22:16,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132871 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:22:22,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:22:22,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107582 s, 3.8 MB/s\n'; <rc> = 0 >Thread-14012::WARNING::2013-03-17 09:22:25,052::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:22:25,059::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:22:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:22:26,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995545 s, 411 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:22:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:22:32,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105679 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:22:35,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:22:35,065::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:22:35,527::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:22:35,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:22:35,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:22:36,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:22:36,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112955 s, 363 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:22:42,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:22:42,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00444166 s, 922 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:22:46,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:22:46,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116068 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:22:52,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:22:52,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803708 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:22:56,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:22:56,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0082063 s, 499 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:02,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:02,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163335 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:06,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:06,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153546 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:12,958::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:23:12,959::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:23:12,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:12,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132219 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:12,997::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132219 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:23:12,998::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:23:12,998::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:23:13,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:23:13,002::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:23:13,003::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:23:13,003::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:23:13,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:23:13,004::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:23:13,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:13,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:13,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:23:13,427::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:23:13,429::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:23:13,430::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:23:13,430::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:23:13,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:13,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:13,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:13,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102777 s, 399 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:16,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:16,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011719 s, 350 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:23,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:23,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794716 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:26,768::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:23:26,770::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:23:26,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00127971 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:26,811::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00127971 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:23:26,811::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:23:26,812::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:23:26,813::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:23:26,815::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:26,815::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:26,816::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:26,817::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:26,817::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:26,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:27,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:27,260::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:27,261::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:23:27,264::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:23:27,265::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:23:27,265::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:23:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:27,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:27,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124643 s, 329 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:33,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114767 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14013::WARNING::2013-03-17 09:23:35,566::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:23:35,572::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:23:37,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:37,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163849 s, 250 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:43,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:43,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920816 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:23:45,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:23:45,576::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:23:46,063::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:23:46,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:23:46,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:23:47,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:47,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013795 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:23:53,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:23:53,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158431 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:23:57,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:23:58,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119659 s, 342 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:24:03,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:24:04,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00953215 s, 430 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:24:08,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:24:08,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773225 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:24:14,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:24:14,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00257619 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:24:18,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:24:18,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187068 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:24:24,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:24:24,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103421 s, 396 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:24:28,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:24:28,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166616 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:24:34,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:24:34,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133514 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:24:38,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:24:38,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124283 s, 330 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:24:44,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:24:44,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116825 s, 351 kB/s\n'; <rc> = 0 >Thread-14014::WARNING::2013-03-17 09:24:46,099::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:24:46,103::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:24:48,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:24:48,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745875 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:24:54,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:24:54,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841497 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:24:56,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:24:56,110::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:24:56,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:24:56,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:24:56,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:24:58,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:24:58,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00780022 s, 525 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:25:04,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:25:04,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727082 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:25:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:25:08,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147818 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:25:14,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:25:14,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137328 s, 298 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:25:18,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:25:18,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920549 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:25:24,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:25:24,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115269 s, 355 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:25:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:25:28,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134178 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:25:34,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:25:34,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279279 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:25:38,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:25:38,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155913 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:25:44,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:25:44,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142436 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:25:48,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:25:48,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900228 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:25:54,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:25:54,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102541 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14015::WARNING::2013-03-17 09:25:56,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:25:56,603::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:25:58,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:25:58,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123336 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:26:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:26:04,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816026 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:26:06,610::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:26:06,611::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:26:07,058::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:26:07,065::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:26:07,066::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:26:09,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:26:09,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146626 s, 279 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:26:14,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:26:14,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868647 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:26:19,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:26:19,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844581 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:26:24,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:26:24,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085426 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:26:29,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:26:29,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101786 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:26:34,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:26:34,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103694 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:26:39,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:26:39,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760149 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:26:44,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:26:44,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895741 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:26:49,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:26:49,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914904 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:26:54,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:26:54,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181992 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:26:59,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:26:59,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145683 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:27:04,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:27:04,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786506 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14016::WARNING::2013-03-17 09:27:07,090::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:27:07,094::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:27:09,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:27:09,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832485 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:27:14,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:27:14,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815435 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:27:17,149::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:27:17,150::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:27:17,151::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:27:17,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:27:17,581::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:27:17,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:27:19,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:27:19,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148599 s, 276 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:27:24,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:27:24,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991446 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:27:29,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:27:29,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174738 s, 234 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:27:34,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:27:34,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131542 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:27:39,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:27:39,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894037 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:27:44,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:27:45,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148842 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:27:49,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:27:49,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008363 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:27:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:27:55,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708553 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:27:59,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:27:59,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138365 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:05,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:05,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0038934 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:28:09,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:09,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891387 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:15,158::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:28:15,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:28:15,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:15,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013729 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:15,207::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013729 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:28:15,208::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:28:15,208::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:28:15,210::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:28:15,211::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:28:15,211::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:28:15,212::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:28:15,212::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:28:15,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:28:15,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:15,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:15,669::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:28:15,670::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:28:15,673::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:28:15,674::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:28:15,675::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:28:15,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:16,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:16,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:16,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220911 s, 185 kB/s\n'; <rc> = 0 >Thread-14017::WARNING::2013-03-17 09:28:17,614::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:28:17,620::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:28:20,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:20,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01295 s, 316 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:26,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:26,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178467 s, 230 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:28:27,623::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:28:27,624::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:28:28,035::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:28:28,041::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:28:28,043::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,133::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:28:30,135::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:28:30,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:30,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0147063 s, 139 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:28:30,180::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0147063 s, 139 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:28:30,180::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:28:30,181::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:28:30,182::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:28:30,186::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,187::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,187::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,188::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,188::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:30,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:28:30,559::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:28:30,560::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:28:30,563::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:28:30,564::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:28:30,564::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:28:30,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:30,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:28:30,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:30,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00403136 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:36,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:36,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851334 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:28:40,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:41,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791391 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:46,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:46,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00983437 s, 416 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:28:51,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:28:51,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813926 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:28:56,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:28:56,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792108 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:29:01,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:29:01,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749675 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:29:06,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:29:06,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894502 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:29:11,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:29:11,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264897 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:29:16,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:29:16,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771785 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:29:21,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:29:21,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157216 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:29:26,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:29:26,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954913 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14018::WARNING::2013-03-17 09:29:28,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:29:28,073::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:29:31,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:29:31,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771038 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:29:36,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:29:36,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825126 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:29:38,080::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:29:38,081::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:29:38,526::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:29:38,533::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:29:38,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:29:41,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:29:41,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201948 s, 203 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:29:46,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:29:46,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016923 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:29:51,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:29:51,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147371 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:29:56,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:29:56,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854281 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:30:01,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:30:01,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307361 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:30:06,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:30:06,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0071507 s, 573 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:30:11,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:30:11,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449222 s, 912 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:30:16,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:30:16,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998205 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:30:21,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:30:21,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133464 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:30:26,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:30:26,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202089 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:30:31,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:30:31,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870766 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:30:36,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:30:36,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872101 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14019::WARNING::2013-03-17 09:30:38,567::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:30:38,572::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:30:41,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:30:42,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114529 s, 358 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:30:46,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:30:47,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777019 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:30:48,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:30:48,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:30:49,007::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:30:49,014::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:30:49,015::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:30:52,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:30:52,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191042 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:30:57,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:30:57,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791973 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:31:02,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:31:02,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876193 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:31:07,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:31:07,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848686 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:31:12,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:31:12,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220015 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:31:17,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:31:17,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860132 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:31:22,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:31:22,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765293 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:31:27,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:31:27,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137897 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:31:32,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:31:32,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196897 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:31:37,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:31:37,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147089 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:31:42,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:31:42,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161831 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:31:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:31:47,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107685 s, 3.8 MB/s\n'; <rc> = 0 >Thread-14020::WARNING::2013-03-17 09:31:49,055::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:31:49,057::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:31:52,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:31:52,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934136 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:31:57,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:31:57,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849013 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:31:59,067::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:31:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:31:59,473::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:31:59,479::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:31:59,481::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:32:02,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:32:02,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138047 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:32:07,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:32:07,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00521017 s, 786 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:32:12,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:32:12,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080973 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:32:17,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:32:17,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969206 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:32:22,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:32:22,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00318985 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:32:27,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:32:27,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827148 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:32:32,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:32:32,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110747 s, 370 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:32:37,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:32:37,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171608 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:32:42,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:32:43,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788821 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:32:47,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:32:47,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764296 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:32:53,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:32:53,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860902 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:32:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:32:57,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981965 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14021::WARNING::2013-03-17 09:32:59,510::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:32:59,514::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:33:03,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:03,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081973 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:07,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:07,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073608 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:33:09,515::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:33:09,516::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:33:09,517::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:33:09,949::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:33:09,956::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:33:09,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:13,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:13,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159878 s, 256 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:17,598::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:33:17,599::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:33:17,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:17,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000885679 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:17,630::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000885679 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:33:17,630::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:33:17,630::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:33:17,632::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:33:17,633::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:33:17,634::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:33:17,634::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:33:17,635::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:33:17,635::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:33:17,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:18,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:18,059::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:33:18,060::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:33:18,062::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:33:18,064::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:33:18,064::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:33:18,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:18,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:18,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:18,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160381 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:23,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:23,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244801 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:28,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:28,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135572 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:33,403::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:33:33,404::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:33:33,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:33,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126469 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:33,447::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126469 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:33:33,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:33:33,448::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:33:33,450::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:33:33,451::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:33,451::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:33,452::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:33,452::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:33,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:33,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:33,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:33,895::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:33:33,896::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:33:33,899::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:33:33,900::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:33:33,901::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:33:33,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:34,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:34,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112711 s, 363 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:38,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:38,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826585 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:44,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:44,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765649 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:48,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:48,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133328 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:33:54,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:33:54,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803812 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:33:58,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:33:58,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154709 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:34:04,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:34:04,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996623 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:34:08,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:34:08,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000676016 s, 6.1 MB/s\n'; <rc> = 0 >Thread-14022::WARNING::2013-03-17 09:34:09,985::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:34:09,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:34:14,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:34:14,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808636 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:34:18,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:34:18,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837113 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:34:19,992::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:34:19,994::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:34:20,434::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:34:20,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:34:20,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:34:24,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:34:24,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906982 s, 452 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:34:28,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:34:28,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144458 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:34:34,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:34:34,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938298 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:34:38,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:34:38,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850688 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:34:44,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:34:44,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819321 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:34:48,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:34:48,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991947 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:34:54,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:34:54,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150154 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:34:58,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:34:58,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155622 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:35:04,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:35:04,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138542 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:35:08,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:35:09,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137044 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:35:15,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:35:15,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140506 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:35:19,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:35:19,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217347 s, 1.9 MB/s\n'; <rc> = 0 >Thread-14023::WARNING::2013-03-17 09:35:20,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:35:20,475::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:35:25,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:35:25,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728939 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:35:29,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:35:29,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079819 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:35:30,484::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:35:30,485::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:35:30,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:35:30,986::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:35:30,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:35:35,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:35:35,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166924 s, 245 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:35:39,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:35:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900539 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:35:45,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:35:45,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164019 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:35:49,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:35:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971161 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:35:55,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:35:55,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783113 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:35:59,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:35:59,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782753 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:36:05,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:36:05,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778047 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:36:09,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:36:09,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145396 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:36:15,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:36:15,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178502 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:36:19,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:36:19,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831759 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:36:25,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:36:25,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290795 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:36:29,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:36:29,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091293 s, 4.5 MB/s\n'; <rc> = 0 >Thread-14024::WARNING::2013-03-17 09:36:31,012::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:36:31,016::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:36:35,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:36:35,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999081 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:36:39,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:36:39,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081733 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:36:41,018::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:36:41,020::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:36:41,437::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:36:41,444::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:36:41,445::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:36:45,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:36:45,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00799055 s, 513 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:36:49,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:36:49,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901378 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:36:55,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:36:55,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131514 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:36:59,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:36:59,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148838 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:37:05,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:37:05,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139123 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:37:09,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:37:09,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987372 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:37:15,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:37:16,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826323 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:37:19,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:37:19,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772591 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:37:26,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:37:26,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949465 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:37:29,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:37:29,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928214 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:37:36,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:37:36,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131218 s, 312 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:37:39,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:37:39,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134383 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14025::WARNING::2013-03-17 09:37:41,470::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:37:41,477::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:37:46,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:37:46,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818483 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:37:49,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:37:49,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785839 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:37:51,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:37:51,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:37:51,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:37:51,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:37:51,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:37:56,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:37:56,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703615 s, 582 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:37:59,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:37:59,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144283 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:06,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:06,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899498 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:09,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814312 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:16,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:16,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993886 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:19,972::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:38:19,974::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:38:19,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:20,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000842284 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:20,003::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000842284 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:38:20,003::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:38:20,004::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:38:20,006::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:38:20,007::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:38:20,007::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:38:20,008::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:38:20,008::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:38:20,008::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:38:20,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:20,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:20,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:38:20,442::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:38:20,445::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:38:20,446::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:38:20,446::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:38:20,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:20,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:20,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:20,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00677289 s, 605 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:26,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:26,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124652 s, 329 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:30,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:30,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178798 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:36,623::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:38:36,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:38:36,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00154227 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:36,666::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00154227 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:38:36,666::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:38:36,667::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:38:36,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:38:36,669::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:38:36,670::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:38:36,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:38:36,671::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:38:36,671::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:38:36,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:37,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:37,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:38:37,147::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:38:37,149::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:38:37,151::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:38:37,151::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:38:37,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:37,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:37,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:37,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00942481 s, 435 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:40,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:40,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892395 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:38:47,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:47,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809107 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:38:51,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:38:51,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829477 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14026::WARNING::2013-03-17 09:38:51,928::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:38:51,932::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:38:57,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:38:57,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819216 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:39:01,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:39:01,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800639 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:39:01,934::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:39:01,936::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:39:01,937::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:39:02,394::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:39:02,401::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:39:02,402::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:39:07,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:39:07,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145552 s, 281 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:39:11,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:39:11,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259029 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:39:17,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:39:17,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872832 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:39:21,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:39:21,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713873 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:39:27,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:39:27,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155735 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:39:31,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:39:31,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165144 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:39:38,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:39:38,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090057 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:39:41,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:39:41,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136908 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:39:48,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:39:48,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830365 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:39:51,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:39:51,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140251 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:39:58,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:39:58,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150686 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:40:01,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:40:01,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0286756 s, 143 kB/s\n'; <rc> = 0 >Thread-14027::WARNING::2013-03-17 09:40:02,434::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:40:02,440::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:40:08,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:40:08,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240467 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:40:11,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:40:11,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148455 s, 276 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:40:12,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:40:12,454::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:40:12,907::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:40:12,914::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:40:12,916::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:40:18,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:40:18,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014548 s, 282 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:40:21,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:40:21,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139531 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:40:28,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:40:28,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017074 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:40:31,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:40:31,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737574 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:40:38,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:40:38,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757236 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:40:41,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:40:41,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137239 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:40:48,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:40:48,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168608 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:40:51,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:40:51,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775497 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:40:58,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:40:58,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223632 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:41:01,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:41:01,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797223 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:41:08,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:41:08,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144356 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:41:11,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:41:12,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210171 s, 195 kB/s\n'; <rc> = 0 >Thread-14028::WARNING::2013-03-17 09:41:12,946::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:41:12,950::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:41:18,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:41:18,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812932 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:41:22,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:41:22,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833901 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:41:22,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:41:22,954::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:41:23,411::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:41:23,418::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:41:23,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:41:28,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:41:28,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179998 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:41:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:41:32,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0272683 s, 150 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:41:38,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:41:38,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797768 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:41:42,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:41:42,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828679 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:41:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:41:49,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135378 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:41:52,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:41:52,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108428 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:41:59,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:41:59,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795041 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:42:02,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:42:02,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138063 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:42:09,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:42:09,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769239 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:42:12,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:42:12,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135176 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:42:19,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:42:19,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100277 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:42:22,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:42:22,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398407 s, 1.0 MB/s\n'; <rc> = 0 >Thread-14029::WARNING::2013-03-17 09:42:23,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:42:23,455::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:42:29,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:42:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796968 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:42:32,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:42:32,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078205 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:42:33,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:42:33,459::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:42:33,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:42:33,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:42:33,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:42:39,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:42:39,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00832356 s, 492 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:42:42,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:42:42,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101192 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:42:49,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:42:49,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779669 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:42:52,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:42:52,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093193 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:42:59,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:42:59,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138468 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:02,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:02,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073189 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:43:09,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:09,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789661 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:12,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:12,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152892 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:43:19,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:19,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140421 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:22,643::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:43:22,645::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:43:22,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:22,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145595 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:22,686::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145595 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:43:22,687::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:43:22,687::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:43:22,689::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:43:22,690::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:43:22,690::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:43:22,691::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:43:22,691::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:43:22,692::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:43:22,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:23,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:23,136::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:43:23,137::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:43:23,140::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:43:23,141::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:43:23,141::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:43:23,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:23,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:23,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:23,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137727 s, 297 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:43:29,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:29,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00565072 s, 725 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:33,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:33,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00482737 s, 848 kB/s\n'; <rc> = 0 >Thread-14030::WARNING::2013-03-17 09:43:33,936::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:43:33,940::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:43:39,917::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:43:39,918::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:43:39,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:39,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000854959 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:43:39,947::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000854959 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:43:39,947::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:43:39,947::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:43:39,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:43:39,950::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:39,951::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:39,951::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:39,952::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:39,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:39,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:40,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:43:40,362::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:40,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:43:40,365::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:43:40,366::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:43:40,367::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:43:40,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:40,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:43:40,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:40,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288104 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:43,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:43,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887491 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:43:43,943::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:43:43,945::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:43:44,406::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:43:44,413::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:43:44,414::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:43:50,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:43:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00812309 s, 504 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:43:53,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:43:53,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087276 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:44:00,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:44:00,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911869 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:44:03,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:44:03,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917126 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:44:10,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:44:10,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00852556 s, 480 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:44:13,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:44:13,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896517 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:44:21,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:44:21,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176818 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:44:23,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:44:23,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797847 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:44:31,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:44:31,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014255 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:44:33,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:44:33,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157049 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:44:41,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:44:41,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164239 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:44:43,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:44:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00333685 s, 1.2 MB/s\n'; <rc> = 0 >Thread-14031::WARNING::2013-03-17 09:44:44,449::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:44:44,453::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:44:51,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:44:51,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828013 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:44:53,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:44:54,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078375 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:44:54,509::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:44:54,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:44:54,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:44:54,941::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:44:54,948::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:44:54,950::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:45:01,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:45:01,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150719 s, 272 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:45:04,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:45:04,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748093 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:45:11,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:45:11,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851623 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:45:14,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:45:14,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00273296 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:45:21,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:45:21,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840672 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:45:24,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:45:24,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727052 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:45:31,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:45:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012998 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:45:34,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:45:34,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810615 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:45:41,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:45:41,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016451 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:45:44,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:45:44,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137362 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:45:51,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:45:51,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146137 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:45:54,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:45:54,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836439 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14032::WARNING::2013-03-17 09:45:54,974::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:45:54,978::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:46:01,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:46:01,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008325 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:46:04,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:46:04,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183352 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:46:04,980::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:46:04,982::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:46:05,398::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:46:05,405::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:46:05,407::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:46:11,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:46:11,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104538 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:46:14,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:46:14,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181639 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:46:21,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:46:22,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178789 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:46:24,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:46:24,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137095 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:46:32,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:46:32,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083241 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:46:34,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:46:34,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013021 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:46:42,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:46:42,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144963 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:46:44,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:46:44,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148587 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:46:52,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:46:52,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155167 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:46:54,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:46:54,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815081 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:47:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:47:02,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804244 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:47:04,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:47:04,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898902 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14033::WARNING::2013-03-17 09:47:05,447::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:47:05,450::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:47:12,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:47:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157798 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:47:14,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:47:15,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741865 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:47:15,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:47:15,455::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:47:15,902::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:47:15,909::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:47:15,911::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:47:22,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:47:22,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:47:25,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:47:25,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170163 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:47:32,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:47:32,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805421 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:47:35,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:47:35,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812463 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:47:42,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:47:42,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137005 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:47:45,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:47:45,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148927 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:47:52,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:47:52,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761288 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:47:55,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:47:55,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139813 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:02,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:02,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783161 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:05,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:05,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741297 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:12,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:12,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145181 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:15,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:15,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143848 s, 285 kB/s\n'; <rc> = 0 >Thread-14034::WARNING::2013-03-17 09:48:15,938::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:48:15,942::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:48:22,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:22,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839713 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:25,410::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:48:25,411::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:48:25,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:25,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148365 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:25,468::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148365 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:48:25,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:48:25,468::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:48:25,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:48:25,471::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:48:25,473::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:48:25,473::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:48:25,474::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:48:25,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:48:25,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-21::DEBUG::2013-03-17 09:48:25,953::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-21::DEBUG::2013-03-17 09:48:25,976::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:26,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:26,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:48:26,051::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:48:26,052::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:48:26,053::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:48:26,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-21::DEBUG::2013-03-17 09:48:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-22::DEBUG::2013-03-17 09:48:26,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:26,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::WARNING::2013-03-17 09:48:26,530::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:48:26,541::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:48:26,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01395 s, 294 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:32,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:32,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00914807 s, 448 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:36,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:36,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794561 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:43,034::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:48:43,036::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:48:43,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:43,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000713356 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:43,068::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000713356 s, 2.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:48:43,068::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:48:43,068::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:48:43,070::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:48:43,071::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:48:43,072::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:48:43,072::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:48:43,072::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:48:43,073::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:48:43,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:43,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:43,492::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:48:43,493::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:48:43,497::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:48:43,498::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:48:43,499::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:48:43,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:43,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:43,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:43,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00967438 s, 423 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:46,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:46,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129345 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:48:53,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:48:54,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0043909 s, 933 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:48:56,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:48:56,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787456 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:49:04,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:49:04,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889935 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:49:06,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:06,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831871 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:49:14,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:49:14,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016764 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:49:16,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:16,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141661 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:49:24,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:49:24,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101718 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14035::WARNING::2013-03-17 09:49:26,585::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:49:26,588::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:49:26,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:26,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117871 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:49:34,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:49:34,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918195 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:49:36,590::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:49:36,591::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:36,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:36,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152488 s, 269 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:49:37,077::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:49:37,083::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:49:37,085::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:49:44,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:49:44,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177089 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:49:46,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:46,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782111 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:49:54,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:49:54,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780734 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:49:56,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:49:56,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130708 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:50:04,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:50:04,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150475 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:50:06,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:07,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0298833 s, 137 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:50:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:50:14,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136476 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:50:17,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:17,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207221 s, 198 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:50:24,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:50:24,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811585 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:50:27,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:27,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142386 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:50:34,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:50:34,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012396 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14036::WARNING::2013-03-17 09:50:37,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:50:37,119::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:50:37,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:37,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171111 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:50:44,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:50:44,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914403 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:50:47,121::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:50:47,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:50:47,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:47,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:47,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199655 s, 205 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:50:47,571::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:50:47,577::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:50:47,579::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:50:54,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:50:54,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179366 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:50:57,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:50:57,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867199 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:51:05,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:51:05,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821671 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:51:07,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:51:07,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188986 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:51:15,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:51:15,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143769 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:51:17,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:51:17,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135039 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:51:25,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:51:25,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000651732 s, 6.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:51:27,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:51:27,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846759 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:51:35,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:51:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016584 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:51:37,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:51:37,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774383 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:51:45,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:51:45,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107693 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:51:47,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:51:47,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744001 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14037::WARNING::2013-03-17 09:51:47,628::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:51:47,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:51:55,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:51:55,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859464 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:51:57,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:51:57,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00679192 s, 603 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:51:57,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:51:57,641::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:51:58,115::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:51:58,122::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:51:58,123::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:52:05,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:52:05,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015695 s, 261 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:52:07,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:52:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886598 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:52:15,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:52:15,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838359 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:52:17,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:52:17,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103453 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:52:25,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:52:25,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145316 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:52:27,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:52:27,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0247347 s, 166 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:52:35,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:52:35,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157856 s, 259 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:52:37,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:52:37,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008266 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:52:45,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:52:45,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145768 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:52:47,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:52:47,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188924 s, 217 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:52:55,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:52:55,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088661 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:52:57,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:52:58,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00241862 s, 1.7 MB/s\n'; <rc> = 0 >Thread-14038::WARNING::2013-03-17 09:52:58,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:52:58,158::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:53:05,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821461 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:08,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:08,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135923 s, 301 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:53:08,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:53:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:53:08,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:53:08,643::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:53:08,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:16,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:16,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172891 s, 237 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:18,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:18,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132559 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:26,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:26,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856772 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:28,209::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:53:28,221::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:53:28,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:28,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137156 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:28,281::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137156 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:53:28,282::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:53:28,282::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:53:28,284::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:53:28,285::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:53:28,285::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:53:28,286::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:53:28,286::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:53:28,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:53:28,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:28,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:28,719::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:53:28,720::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:53:28,722::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:53:28,724::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:53:28,724::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:53:28,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:29,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:29,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:29,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195257 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:36,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:36,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926671 s, 442 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:39,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:39,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743113 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:46,316::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:53:46,318::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:53:46,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00139528 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:46,363::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00139528 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:53:46,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:53:46,363::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:53:46,365::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:53:46,366::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:46,367::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:46,367::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:46,367::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:46,368::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:46,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:46,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:46,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:53:46,788::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:53:46,790::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:53:46,791::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:53:46,792::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:53:46,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:47,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:47,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:47,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129177 s, 317 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:49,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:49,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103661 s, 395 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:53:57,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:53:57,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157824 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:53:59,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:53:59,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001439 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:54:07,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:54:07,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134487 s, 305 kB/s\n'; <rc> = 0 >Thread-14039::WARNING::2013-03-17 09:54:08,673::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:54:08,678::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:54:09,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:54:09,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161557 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:54:17,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:54:17,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162611 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:54:18,682::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:54:18,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:54:19,116::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:54:19,123::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:54:19,124::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:54:19,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:54:19,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176693 s, 232 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:54:27,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:54:28,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156229 s, 262 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:54:29,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:54:29,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078056 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:54:38,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:54:38,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090318 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:54:39,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:54:39,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931082 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:54:48,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:54:48,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00353042 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:54:49,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:54:49,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148335 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:54:58,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:54:58,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802609 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:54:59,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:54:59,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155805 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:55:08,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:55:08,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907005 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:55:09,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:55:09,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780679 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:55:18,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:55:18,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828403 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14040::WARNING::2013-03-17 09:55:19,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:55:19,162::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:55:19,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:55:19,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252774 s, 162 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:55:28,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:55:28,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0322837 s, 127 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:55:29,164::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:55:29,166::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:55:29,562::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:55:29,569::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:55:29,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:55:29,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:55:29,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203288 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:55:38,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:55:38,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808879 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:55:39,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:55:39,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961405 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:55:48,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:55:48,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146946 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:55:49,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:55:49,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140957 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:55:58,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:55:58,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805737 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:55:59,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:55:59,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167412 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:56:08,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:56:09,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732621 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:56:09,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:56:09,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770195 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:56:19,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:56:19,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174843 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:56:19,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:56:19,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889991 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:56:29,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:56:29,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128745 s, 3.2 MB/s\n'; <rc> = 0 >Thread-14041::WARNING::2013-03-17 09:56:29,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:56:29,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:56:29,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:56:30,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819465 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:56:39,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:56:39,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108206 s, 379 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:56:39,601::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 09:56:39,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:56:39,604::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 09:56:40,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 09:56:40,050::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:56:40,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:56:40,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:56:40,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173888 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:56:49,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:56:49,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766411 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:56:50,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:56:50,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854424 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:56:59,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:56:59,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916389 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:57:00,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:00,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868679 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:57:09,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:57:09,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144809 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:57:10,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:10,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805975 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:57:19,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:57:19,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134378 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:57:20,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:20,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867377 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:57:29,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:57:29,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777792 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:57:30,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:30,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934221 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:57:39,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:57:39,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137784 s, 297 kB/s\n'; <rc> = 0 >Thread-14042::WARNING::2013-03-17 09:57:40,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:57:40,101::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 09:57:40,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:40,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206844 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:57:50,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:57:50,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109605 s, 374 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:57:50,108::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:57:50,117::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:50,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:57:50,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181416 s, 226 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:57:50,614::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:57:50,620::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:57:50,621::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:00,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:00,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150099 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:00,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:00,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124749 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:10,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:10,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885541 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:10,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:10,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827778 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:20,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:20,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181274 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:20,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:20,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165175 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:30,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:30,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01502 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:30,516::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 09:58:30,518::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:58:30,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:30,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121709 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:30,582::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121709 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 09:58:30,584::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 09:58:30,584::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 09:58:30,589::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 09:58:30,596::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:58:30,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:58:30,602::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:58:30,605::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:58:30,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 09:58:30,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:31,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:31,061::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:58:31,062::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 09:58:31,064::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 09:58:31,066::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 09:58:31,066::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 09:58:31,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:31,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:31,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:31,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159859 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:40,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:40,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208427 s, 197 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:58:41,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:41,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165373 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14043::WARNING::2013-03-17 09:58:50,660::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 09:58:50,666::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 09:58:50,766::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 09:58:50,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:58:50,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:50,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0189117 s, 108 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:50,828::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0189117 s, 108 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 09:58:50,829::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 09:58:50,829::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 09:58:50,831::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 09:58:50,832::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:50,832::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:50,835::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:50,836::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:50,836::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:50,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:51,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:51,257::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 09:58:51,258::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 09:58:51,260::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 09:58:51,262::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 09:58:51,262::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 09:58:51,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:58:51,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175281 s, 234 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:51,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:58:51,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:58:51,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00860372 s, 476 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 09:59:00,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 09:59:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 09:59:01,090::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 09:59:01,097::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 09:59:01,099::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 09:59:01,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:59:01,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154396 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:59:01,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:59:01,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161633 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:59:11,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:59:11,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145209 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:59:11,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:59:11,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797362 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:59:21,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:59:21,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819223 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:59:21,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:59:22,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145833 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:59:31,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:59:31,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713613 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:59:32,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:59:32,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848211 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:59:41,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:59:41,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133383 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:59:42,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:59:42,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792195 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 09:59:51,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 09:59:51,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.003402 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 09:59:52,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 09:59:52,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145692 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14044::WARNING::2013-03-17 10:00:01,122::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:00:01,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:00:02,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:00:02,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876146 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:00:02,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:00:02,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839543 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:00:11,128::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:00:11,129::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:00:11,556::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:00:11,563::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:00:11,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:00:12,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:00:12,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117094 s, 350 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:00:12,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:00:12,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170548 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:00:22,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:00:22,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896178 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:00:22,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:00:22,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00385431 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:00:32,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:00:32,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000688859 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:00:32,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:00:32,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155315 s, 264 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:00:42,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:00:42,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080584 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:00:42,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:00:42,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113695 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:00:52,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:00:52,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136481 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:00:52,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:00:52,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177643 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:01:02,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:01:02,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941746 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:01:02,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:01:02,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138634 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14045::WARNING::2013-03-17 10:01:11,590::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:01:11,593::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:01:12,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:01:12,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777469 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:01:12,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:01:12,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00514977 s, 795 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:01:21,595::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:01:21,597::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:01:22,029::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:01:22,035::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:01:22,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:01:22,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:01:22,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177174 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:01:22,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:01:23,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156281 s, 262 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:01:32,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:01:32,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735338 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:01:33,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:01:33,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761017 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:01:42,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:01:42,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840148 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:01:43,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:01:43,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015494 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:01:52,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:01:52,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843174 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:01:53,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:01:53,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00433621 s, 945 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:02:02,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:02:02,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733209 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:02:03,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:02:03,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142417 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:02:12,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:02:12,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796649 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:02:13,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:02:13,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155566 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14046::WARNING::2013-03-17 10:02:22,065::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:02:22,069::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:02:22,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:02:22,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103185 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:02:23,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:02:23,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872365 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:02:32,070::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:02:32,072::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:02:32,073::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:02:32,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:02:32,508::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:02:32,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:02:32,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:02:32,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167814 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:02:33,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:02:33,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135664 s, 302 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:02:42,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:02:42,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137462 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:02:43,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:02:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140847 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:02:52,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:02:52,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088085 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:02:53,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:02:53,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831225 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:02,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:02,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203225 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:03,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:03,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186933 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:12,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:12,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836953 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:13,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:13,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764993 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:22,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:22,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859067 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:23,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:24,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014622 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14047::WARNING::2013-03-17 10:03:32,535::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:03:32,538::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:03:32,901::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:03:32,902::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:03:32,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:32,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000816883 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:32,928::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000816883 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:03:32,928::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:03:32,929::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:03:32,930::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:03:32,931::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:32,932::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:32,933::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:32,933::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:32,933::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:32,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:33,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:33,337::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:33,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:03:33,341::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:03:33,343::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:03:33,343::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:03:33,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:33,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:33,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015741 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:34,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:34,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145086 s, 282 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:03:42,540::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:03:42,542::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:03:43,023::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:03:43,034::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:03:43,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:03:43,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:43,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128042 s, 320 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:44,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:44,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016441 s, 249 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:03:53,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:03:53,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115229 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:54,237::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:03:54,238::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:03:54,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:54,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000781179 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:54,265::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000781179 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:03:54,266::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:03:54,266::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:03:54,268::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:03:54,269::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:03:54,269::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:03:54,270::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:03:54,270::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:03:54,270::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:03:54,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:54,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:54,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:03:54,688::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:03:54,690::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:03:54,691::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:03:54,692::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:03:54,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:55,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:03:55,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:03:55,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00907614 s, 451 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:04:03,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:04:03,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847524 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:04:05,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:04:05,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782389 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:04:13,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:04:13,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143846 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:04:15,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:04:15,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765931 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:04:23,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:04:23,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137212 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:04:25,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:04:25,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735432 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:04:33,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:04:34,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147024 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:04:35,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:04:35,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116971 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14048::WARNING::2013-03-17 10:04:43,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:04:43,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:04:44,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:04:44,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00976385 s, 420 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:04:45,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:04:45,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078243 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:04:53,074::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:04:53,075::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:04:53,478::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:04:53,484::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:04:53,486::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:04:54,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:04:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124296 s, 330 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:04:55,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:04:55,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926804 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:05:04,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:05:04,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772268 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:05:05,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:05:05,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014908 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:05:14,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:05:14,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934333 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:05:15,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:05:15,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728083 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:05:24,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:05:24,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826022 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:05:25,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:05:25,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178805 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:05:34,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:05:34,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144183 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:05:35,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:05:35,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157701 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:05:44,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:05:44,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02726 s, 150 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:05:45,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:05:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145531 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14049::WARNING::2013-03-17 10:05:53,516::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:05:53,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:05:54,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:05:54,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863362 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:05:56,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:05:56,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818627 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:06:03,532::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:06:03,534::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:06:03,962::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:06:03,969::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:06:03,970::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:06:04,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:06:04,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017242 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:06:06,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:06:06,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108446 s, 378 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:06:14,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:06:14,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868812 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:06:16,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:06:16,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140965 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:06:24,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:06:24,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014493 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:06:26,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:06:26,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015198 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:06:34,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:06:34,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170421 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:06:36,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:06:36,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727224 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:06:44,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:06:44,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956497 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:06:46,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:06:46,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791505 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:06:54,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:06:54,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807138 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:06:56,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:06:56,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883931 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14050::WARNING::2013-03-17 10:07:03,998::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:07:04,001::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:07:04,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:07:04,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143582 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:07:06,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:07:06,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870369 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:07:14,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:07:14,004::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:07:14,405::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:07:14,412::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:07:14,413::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:07:14,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:07:14,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115158 s, 356 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:07:16,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:07:16,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154535 s, 265 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:07:24,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:07:24,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146956 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:07:26,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:07:26,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142553 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:07:34,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:07:34,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791194 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:07:36,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:07:36,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088542 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:07:44,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:07:44,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014835 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:07:46,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:07:46,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00733836 s, 558 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:07:54,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:07:54,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087126 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:07:56,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:07:57,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182622 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:04,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:04,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144108 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:07,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:07,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971645 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14051::WARNING::2013-03-17 10:08:14,444::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:08:14,447::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:08:14,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:14,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104199 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:17,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:17,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843162 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:08:24,448::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:08:24,449::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:08:24,450::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:08:24,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:08:24,926::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:08:24,928::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:24,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:25,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135973 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:27,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:27,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173219 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:35,063::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:08:35,064::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:08:35,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:35,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000901748 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:35,092::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000901748 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:08:35,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:08:35,093::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:08:35,094::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:08:35,095::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:35,096::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:35,096::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:35,097::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:35,097::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:35,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:35,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:35,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:08:35,543::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:08:35,545::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:08:35,546::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:08:35,547::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:08:35,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:35,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:35,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:35,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124199 s, 330 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:37,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:37,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133913 s, 306 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:45,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:46,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137796 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:47,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:47,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743978 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:08:56,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:08:56,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131657 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:57,461::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:08:57,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:08:57,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:57,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132112 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:57,503::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132112 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:08:57,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:08:57,504::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:08:57,506::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:08:57,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:08:57,507::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:08:57,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:08:57,508::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:08:57,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:08:57,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:57,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:57,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:08:57,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:08:57,925::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:08:57,926::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:08:57,926::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:08:57,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:58,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:08:58,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:08:58,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00249311 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:09:06,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:09:06,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756167 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:09:08,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:09:08,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093907 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:09:16,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:09:16,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073304 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:09:18,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:09:18,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872075 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14052::WARNING::2013-03-17 10:09:24,953::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:09:24,957::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:09:26,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:09:26,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951805 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:09:28,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:09:28,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078318 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:09:34,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:09:34,960::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:09:35,361::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:09:35,368::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:09:35,369::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:09:36,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:09:36,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121315 s, 338 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:09:38,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:09:38,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104683 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:09:46,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:09:46,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146214 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:09:48,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:09:48,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855695 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:09:56,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:09:56,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871724 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:09:58,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:09:58,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769895 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:10:06,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:10:06,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792686 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:10:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:10:08,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128309 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:10:16,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:10:16,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766546 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:10:18,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:10:18,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981725 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:10:26,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:10:26,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814108 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:10:28,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:10:29,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157276 s, 260 kB/s\n'; <rc> = 0 >Thread-14053::WARNING::2013-03-17 10:10:35,399::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:10:35,403::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:10:36,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:10:36,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815819 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:10:39,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:10:39,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916481 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:10:45,405::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:10:45,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:10:45,834::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:10:45,840::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:10:45,842::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:10:46,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:10:46,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135886 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:10:49,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:10:49,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101266 s, 404 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:10:56,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:10:56,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:10:59,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:10:59,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187785 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:11:06,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:11:06,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800921 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:11:09,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:11:09,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127575 s, 321 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:11:16,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:11:16,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146313 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:11:19,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:11:19,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079236 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:11:26,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:11:26,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937983 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:11:29,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:11:29,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151719 s, 270 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:11:36,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:11:36,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148939 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:11:39,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:11:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843107 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14054::WARNING::2013-03-17 10:11:45,872::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:11:45,876::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:11:46,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:11:46,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0038163 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:11:49,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:11:49,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185528 s, 221 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:11:55,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:11:55,880::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:11:56,285::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:11:56,291::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:11:56,293::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:11:56,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:11:56,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105622 s, 388 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:11:59,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:11:59,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01249 s, 328 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:12:06,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:12:06,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307782 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:12:09,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:12:09,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140359 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:12:16,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:12:16,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853865 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:12:19,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:12:19,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864402 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:12:26,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:12:26,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862854 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:12:29,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:12:30,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133081 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:12:36,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:12:36,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793027 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:12:40,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:12:40,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138275 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:12:46,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:12:47,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097106 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:12:50,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:12:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00654249 s, 626 kB/s\n'; <rc> = 0 >Thread-14055::WARNING::2013-03-17 10:12:56,321::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:12:56,327::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:12:57,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:12:57,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805896 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:13:00,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:13:00,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891172 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:13:06,330::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:13:06,331::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:13:06,747::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:13:06,753::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:13:06,755::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:07,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:07,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019846 s, 206 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:13:10,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:13:10,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00623399 s, 657 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:17,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:17,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084851 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:13:20,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:13:20,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141022 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:27,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:27,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140473 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:13:30,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:13:30,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135311 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:37,231::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:13:37,232::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:13:37,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:37,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000946176 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:37,264::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000946176 s, 2.2 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:13:37,264::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:13:37,265::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:13:37,266::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:13:37,267::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:37,268::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:37,268::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:37,269::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:37,269::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:37,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:37,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:37,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:13:37,702::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:13:37,705::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:13:37,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:13:37,706::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:13:37,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:38,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:38,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:38,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122621 s, 334 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:13:40,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:13:40,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918668 s, 446 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:48,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:48,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016738 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:13:50,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:13:50,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111674 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:13:58,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:13:58,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874549 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:00,701::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:14:00,702::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:14:00,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:00,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149698 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:00,742::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149698 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:14:00,742::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:14:00,743::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:14:00,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:14:00,749::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:14:00,750::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:14:00,750::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:14:00,751::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:14:00,751::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:14:00,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:01,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:01,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:14:01,234::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:14:01,249::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:14:01,250::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:14:01,251::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:14:01,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:01,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:01,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:01,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00914397 s, 448 kB/s\n'; <rc> = 0 >Thread-14056::WARNING::2013-03-17 10:14:06,784::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:14:06,786::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:14:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:14:08,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818149 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:11,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:11,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147786 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:14:16,788::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:14:16,789::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:14:16,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:14:17,261::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:14:17,268::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:14:17,270::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:14:18,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:14:18,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176472 s, 232 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:21,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:21,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00971554 s, 422 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:14:28,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:14:28,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159563 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:31,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:31,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854094 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:14:38,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:14:38,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141517 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:42,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:42,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789128 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:14:48,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:14:48,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130345 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:14:52,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:14:52,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781729 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:14:58,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:14:58,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151716 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:15:02,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:15:02,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159293 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:15:08,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:15:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816472 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:15:12,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:15:12,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127244 s, 3.2 MB/s\n'; <rc> = 0 >Thread-14057::WARNING::2013-03-17 10:15:17,296::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:15:17,300::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:15:18,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:15:18,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117999 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:15:22,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:15:22,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306793 s, 134 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:15:27,302::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:15:27,303::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:15:27,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:15:27,732::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:15:27,733::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:15:28,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:15:28,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127898 s, 320 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:15:32,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:15:32,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769531 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:15:38,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:15:38,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110323 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:15:42,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:15:42,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154363 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:15:48,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:15:48,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167146 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:15:52,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:15:52,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768626 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:15:58,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:15:58,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141466 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:16:02,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:16:02,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993637 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:16:08,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:16:08,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00293639 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:16:12,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:16:12,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855726 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:16:18,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:16:18,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00387333 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:16:22,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:16:22,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014724 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14058::WARNING::2013-03-17 10:16:27,760::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:16:27,764::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:16:28,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:16:28,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795648 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:16:32,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:16:32,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.039304 s, 104 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:16:37,768::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:16:37,769::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:16:38,172::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:16:38,179::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:16:38,180::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:16:38,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:16:38,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185238 s, 221 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:16:42,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:16:43,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142579 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:16:48,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:16:49,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916621 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:16:53,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:16:53,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847508 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:16:59,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:16:59,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815414 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:17:03,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:17:03,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143902 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:17:09,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:17:09,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168822 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:17:13,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:17:13,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137005 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:17:19,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:17:19,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139046 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:17:23,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:17:23,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143646 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:17:29,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:17:29,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083021 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:17:33,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:17:33,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858558 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14059::WARNING::2013-03-17 10:17:38,210::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:17:38,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:17:39,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:17:39,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764557 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:17:43,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:17:43,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074885 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:17:48,220::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:17:48,221::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:17:48,651::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:17:48,657::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:17:48,659::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:17:49,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:17:49,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191748 s, 214 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:17:53,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:17:53,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850408 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:17:59,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:17:59,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137886 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:18:03,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:18:03,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894794 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:09,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:09,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119828 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:18:13,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:18:13,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807101 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:19,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801604 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:18:23,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:18:23,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202836 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:29,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:29,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748453 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:18:33,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:18:33,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146013 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:39,466::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:18:39,467::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:18:39,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:39,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136895 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:39,504::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136895 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:18:39,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:18:39,504::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:18:39,506::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:18:39,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:18:39,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:18:39,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:18:39,508::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:18:39,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:18:39,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:39,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:39,936::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:18:39,937::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:18:39,940::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:18:39,941::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:18:39,941::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:18:39,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:40,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:18:40,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:40,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181246 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:18:43,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:18:43,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00403601 s, 1.0 MB/s\n'; <rc> = 0 >Thread-14060::WARNING::2013-03-17 10:18:48,695::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:18:48,701::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:18:50,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:18:50,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139836 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:18:53,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:18:54,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0273719 s, 150 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:18:58,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:18:58,705::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:18:59,123::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:18:59,130::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:18:59,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:19:00,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:19:00,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129347 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:04,077::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:19:04,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:19:04,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:04,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000826232 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:04,110::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000826232 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:19:04,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:19:04,111::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:19:04,112::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:19:04,113::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:19:04,114::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:19:04,114::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:19:04,115::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:19:04,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:19:04,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:04,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:04,515::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:19:04,516::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:19:04,518::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:19:04,519::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:19:04,520::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:19:04,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:04,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:04,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:05,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00383133 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:19:10,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:19:10,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785471 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:15,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:15,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008894 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:19:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:19:20,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226293 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:25,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:25,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145005 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:19:30,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:19:30,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137851 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:35,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998026 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:19:40,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:19:40,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0066829 s, 613 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:45,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:45,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165699 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:19:50,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:19:50,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721311 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:19:55,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:19:55,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799065 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14061::WARNING::2013-03-17 10:19:59,161::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:19:59,168::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:20:00,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:20:00,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160906 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:20:05,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:20:05,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776461 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:20:09,170::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:20:09,171::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:20:09,172::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:20:09,616::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:20:09,623::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:20:09,624::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:20:10,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:20:10,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155662 s, 263 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:20:15,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:20:15,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778565 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:20:20,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:20:20,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132719 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:20:25,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:20:25,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177694 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:20:30,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:20:30,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144818 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:20:35,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:20:35,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120667 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:20:40,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:20:40,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808605 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:20:45,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:20:45,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158128 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:20:50,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:20:50,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083993 s, 488 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:20:55,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:20:55,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938455 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:21:00,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:21:01,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810413 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:21:05,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:21:05,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154351 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14062::WARNING::2013-03-17 10:21:09,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:21:09,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:21:11,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:21:11,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141966 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:21:16,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:21:16,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904204 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:21:19,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:21:19,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:21:20,096::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:21:20,104::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:21:20,105::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:21:21,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:21:21,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140562 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:21:26,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:21:26,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868401 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:21:31,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:21:31,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147821 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:21:36,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:21:36,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141572 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:21:41,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:21:41,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818114 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:21:46,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:21:46,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000712447 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:21:51,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:21:51,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757828 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:21:56,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:21:56,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073565 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:22:01,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:22:01,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276162 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:22:06,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:22:06,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00245021 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:22:11,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:22:11,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769516 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:22:16,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:22:16,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175038 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14063::WARNING::2013-03-17 10:22:20,134::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:22:20,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:22:21,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:22:21,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801273 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:22:26,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:22:26,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824548 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:22:30,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:22:30,139::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:22:30,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:22:30,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:22:30,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:22:31,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:22:31,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151294 s, 271 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:22:36,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:22:36,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859079 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:22:41,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:22:41,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814095 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:22:46,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:22:46,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160372 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:22:51,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:22:51,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161407 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:22:56,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:22:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167222 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:01,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:01,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138293 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:23:06,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:23:06,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180871 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:11,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139303 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:23:16,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:23:16,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868786 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:21,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:21,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151224 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:23:26,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:23:27,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162042 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14064::WARNING::2013-03-17 10:23:30,623::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:23:30,627::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:23:31,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:31,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089379 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:23:37,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:23:37,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104272 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:23:40,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:23:40,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:23:41,077::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:23:41,084::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:23:41,085::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:41,716::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:23:41,719::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:23:41,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:41,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00992727 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:41,765::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00992727 s, 206 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:23:41,766::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:23:41,766::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:23:41,768::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:23:41,769::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:41,769::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:41,770::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:41,770::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:41,771::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:41,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:42,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:42,162::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:23:42,163::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:23:42,166::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:23:42,167::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:23:42,167::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:23:42,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:42,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:42,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:42,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000674846 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:23:47,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:23:47,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754331 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:23:52,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:23:52,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010481 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:23:57,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:23:57,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789554 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:24:02,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:24:02,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113043 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:07,285::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:24:07,286::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:24:07,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:07,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00110946 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:07,322::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00110946 s, 1.8 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:24:07,323::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:24:07,323::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:24:07,325::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:24:07,326::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:24:07,326::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:24:07,327::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:24:07,327::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:24:07,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:24:07,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:07,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:07,840::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:24:07,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:24:07,843::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:24:07,844::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:24:07,845::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:24:07,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:08,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:08,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00845936 s, 484 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:24:12,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:24:12,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213374 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:18,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:18,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992555 s, 413 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:24:22,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:24:22,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231911 s, 177 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:28,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:28,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151853 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:24:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:24:32,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145672 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:38,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:38,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130088 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14065::WARNING::2013-03-17 10:24:41,110::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:24:41,114::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:24:42,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:24:42,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876757 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:48,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:48,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749244 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:24:51,116::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:24:51,118::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:24:51,567::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:24:51,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:24:51,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:24:52,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:24:52,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155843 s, 263 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:24:58,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:24:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770052 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:25:02,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:25:03,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139687 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:25:08,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:25:08,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792993 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:25:13,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:25:13,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758914 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:25:18,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:25:18,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776528 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:25:23,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:25:23,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962945 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:25:28,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:25:28,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151913 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:25:33,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:25:33,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016223 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:25:38,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:25:39,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795639 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:25:43,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:25:43,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141176 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:25:49,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:25:49,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785186 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14066::WARNING::2013-03-17 10:25:51,605::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:25:51,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:25:53,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:25:53,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790256 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:25:59,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:25:59,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770994 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:26:01,612::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:26:01,613::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:26:01,614::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:26:02,092::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:26:02,100::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:26:02,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:26:03,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:26:03,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172012 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:26:09,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:26:09,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988944 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:26:13,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:26:13,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110688 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:26:19,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:26:19,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158762 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:26:23,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:26:23,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141006 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:26:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:26:29,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775983 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:26:33,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:26:33,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398777 s, 1.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:26:39,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:26:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134793 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:26:43,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:26:43,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806647 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:26:49,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:26:49,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014229 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:26:53,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:26:53,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125191 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:26:59,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:26:59,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923339 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14067::WARNING::2013-03-17 10:27:02,139::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:27:02,144::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:27:03,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:27:03,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100843 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:27:09,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:27:09,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909967 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:27:12,147::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:27:12,148::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:27:12,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:27:12,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:27:12,631::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:27:13,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:27:13,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268108 s, 153 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:27:19,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:27:19,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899838 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:27:23,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:27:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810149 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:27:29,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:27:29,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794763 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:27:33,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:27:33,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126457 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:27:39,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:27:40,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162851 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:27:43,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:27:43,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138241 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:27:50,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:27:50,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936312 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:27:53,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:27:53,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153266 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:28:00,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:28:00,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818909 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:03,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:03,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144445 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:28:10,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:28:10,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168492 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14068::WARNING::2013-03-17 10:28:12,658::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:28:12,662::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:28:13,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190086 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:28:20,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:28:20,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785038 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:28:22,664::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:28:22,665::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:28:23,095::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:28:23,103::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:28:23,104::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:23,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:24,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251372 s, 163 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:28:30,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:28:30,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786056 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:34,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:34,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016293 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:28:40,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:28:40,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162173 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:44,119::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:28:44,120::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:28:44,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:44,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000817323 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:44,152::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000817323 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:28:44,153::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:28:44,153::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:28:44,155::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:28:44,156::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:44,156::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:44,157::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:44,157::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:44,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:44,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:44,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:44,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:28:44,613::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:28:44,616::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:28:44,617::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:28:44,617::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:28:44,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:44,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:44,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:45,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183211 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:28:50,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:28:50,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128242 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:28:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:28:55,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865052 s, 473 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:00,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:00,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691169 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:29:05,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:29:05,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141557 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:10,761::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:29:10,763::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:29:10,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:10,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144705 s, 142 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:10,817::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144705 s, 142 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:29:10,818::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:29:10,818::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:29:10,820::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:29:10,821::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:29:10,821::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:29:10,822::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:29:10,823::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:29:10,823::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:29:10,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:11,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:11,242::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:29:11,243::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:29:11,245::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:29:11,246::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:29:11,247::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:29:11,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:11,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:11,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:11,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00972685 s, 421 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:29:15,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:29:15,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00823386 s, 497 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:21,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:21,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861473 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14069::WARNING::2013-03-17 10:29:23,136::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:29:23,140::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:29:25,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:29:25,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851103 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:31,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:32,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877669 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:29:33,142::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:29:33,144::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:29:33,591::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:29:33,597::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:29:33,599::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:29:35,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:29:35,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016772 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:42,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:42,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896873 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:29:45,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:29:45,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841793 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:29:52,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:29:52,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149974 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:29:55,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:29:55,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143546 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:30:02,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:30:02,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154035 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:30:05,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:30:05,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829382 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:30:12,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:30:12,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133237 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:30:15,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:30:15,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141295 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:30:22,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:30:22,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780563 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:30:25,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:30:25,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008737 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:30:32,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:30:32,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0302394 s, 135 kB/s\n'; <rc> = 0 >Thread-14070::WARNING::2013-03-17 10:30:33,631::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:30:33,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:30:35,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:30:35,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151519 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:30:42,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:30:43,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896495 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:30:43,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:30:43,647::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:30:44,109::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:30:44,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:30:44,117::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:30:45,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:30:45,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192191 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:30:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:30:53,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901406 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:30:55,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:30:55,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857649 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:31:03,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:31:03,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157647 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:31:05,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:31:05,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232424 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:31:13,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:31:13,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155338 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:31:15,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:31:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780574 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:31:23,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:31:23,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156902 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:31:25,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:31:25,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014921 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:31:33,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:31:33,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728301 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:31:35,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:31:35,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959373 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:31:43,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:31:43,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802378 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14071::WARNING::2013-03-17 10:31:44,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:31:44,154::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:31:45,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:31:45,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831399 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:31:53,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:31:53,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869348 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:31:54,156::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:31:54,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:31:54,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:31:54,610::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:31:54,617::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:31:54,618::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:31:55,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:31:55,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131671 s, 311 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:32:03,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:32:03,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155873 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:32:05,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:32:05,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082291 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:32:13,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:32:13,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263945 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:32:15,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:32:15,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923684 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:32:24,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:32:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117325 s, 349 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:32:25,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:32:26,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144205 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:32:34,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:32:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743822 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:32:36,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:32:36,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013329 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:32:44,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:32:44,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811672 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:32:46,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:32:46,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135846 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:32:54,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:32:54,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259835 s, 1.6 MB/s\n'; <rc> = 0 >Thread-14072::WARNING::2013-03-17 10:32:54,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:32:54,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:32:56,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:32:56,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759921 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:33:04,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:33:04,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758039 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:33:04,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:33:04,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:33:05,173::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:33:05,181::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:33:05,183::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:06,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:06,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190303 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:33:14,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:33:14,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129804 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:16,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:16,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196952 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:33:24,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:33:24,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855529 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:26,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:26,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164061 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:33:34,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:33:34,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015339 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:36,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:36,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809159 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:33:44,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:33:44,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803955 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:46,336::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:33:46,337::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:33:46,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00073007 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:46,372::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00073007 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:33:46,372::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:33:46,373::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:33:46,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:33:46,375::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:46,376::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:46,376::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:46,377::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:46,377::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:46,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:46,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:46,814::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:33:46,815::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:33:46,817::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:33:46,818::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:33:46,819::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:33:46,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:47,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:47,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:47,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141121 s, 290 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:33:54,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:33:55,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154156 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:33:57,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:33:57,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014256 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:05,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:05,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875213 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14073::WARNING::2013-03-17 10:34:05,214::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:34:05,226::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:34:07,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:34:07,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859197 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:34:15,232::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:34:15,234::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:15,259::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:34:15,262::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:34:15,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:15,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0136641 s, 150 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:15,321::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0136641 s, 150 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:34:15,322::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:34:15,322::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:34:15,324::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:34:15,325::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-21::DEBUG::2013-03-17 10:34:15,719::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:34:15,727::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:34:15,727::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:15,732::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:15,733::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:15,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:15,734::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:15,734::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:15,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:16,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:16,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:34:16,149::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:34:16,152::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:34:16,153::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:34:16,153::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:34:16,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:16,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:16,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112395 s, 364 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:34:17,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:34:17,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100632 s, 407 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:26,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815711 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:34:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:34:27,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104258 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:36,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:36,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872773 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:34:37,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:34:37,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810435 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:46,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:46,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000670252 s, 6.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:34:47,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:34:47,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146136 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:34:56,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:34:56,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791951 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:34:57,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:34:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144581 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:35:06,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:35:06,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793931 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:35:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:35:07,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140938 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14074::WARNING::2013-03-17 10:35:15,814::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:35:15,817::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 10:35:16,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:35:16,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790843 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:35:17,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:35:17,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826923 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:35:25,819::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:35:25,820::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:35:26,225::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:35:26,231::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:35:26,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:35:26,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:35:27,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00882487 s, 464 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:35:27,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:35:27,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115276 s, 355 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:35:37,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:35:37,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081478 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:35:37,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:35:37,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156246 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:35:47,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:35:47,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831281 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:35:47,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:35:47,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100412 s, 408 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:35:57,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:35:57,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736884 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:35:57,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:35:57,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942955 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:36:07,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:36:07,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306068 s, 134 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:36:07,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:36:07,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014095 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:36:17,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:36:17,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832216 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:36:17,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:36:17,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307013 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14075::WARNING::2013-03-17 10:36:26,258::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:36:26,262::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 10:36:27,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:36:27,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154253 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:36:27,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:36:27,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852912 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:36:36,277::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:36:36,278::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:36:36,688::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:36:36,695::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:36:36,696::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:36:37,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:36:37,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138056 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:36:37,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:36:38,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188072 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:36:47,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:36:47,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810887 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:36:48,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:36:48,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213057 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:36:57,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:36:57,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145318 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:36:58,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:36:58,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176872 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:37:07,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:37:07,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015241 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:37:08,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:37:08,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797797 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:37:17,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:37:17,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152541 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:37:18,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:37:18,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105303 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:37:27,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:37:27,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866153 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:37:28,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:37:28,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120461 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14076::WARNING::2013-03-17 10:37:36,722::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:37:36,726::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 10:37:38,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:37:38,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812577 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:37:38,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:37:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789458 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:37:46,728::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:37:46,729::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:37:46,729::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:37:47,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:37:47,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:37:47,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:37:48,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:37:48,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254945 s, 161 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:37:48,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:37:48,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100493 s, 408 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:37:58,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:37:58,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957537 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:37:58,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:37:58,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107786 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:38:08,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:38:08,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762233 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:08,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017651 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:38:18,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:38:18,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00065709 s, 6.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:18,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:18,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136963 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:38:28,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:38:28,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145068 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:28,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00239799 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:38:38,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:38:38,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858519 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:38,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:38,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121343 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14077::WARNING::2013-03-17 10:38:47,244::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:38:47,250::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 10:38:48,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:38:48,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00590543 s, 694 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:48,774::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:38:48,776::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:38:48,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:48,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00204491 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:48,842::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00204491 s, 1.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:38:48,843::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:38:48,843::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:38:48,844::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:38:48,845::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:38:48,846::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:38:48,847::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:38:48,847::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:38:48,847::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:38:48,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:49,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:49,322::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:38:49,323::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:38:49,325::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:38:49,326::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:38:49,327::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:38:49,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:49,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:49,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173621 s, 236 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:38:57,258::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:38:57,259::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:38:57,740::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:38:57,747::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:38:57,749::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:38:58,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:38:58,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111892 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:38:59,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:38:59,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136274 s, 301 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:08,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:08,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100312 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:39:09,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:39:09,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840678 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:18,806::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:39:18,807::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:39:18,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:18,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000729802 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:18,838::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000729802 s, 2.8 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:39:18,838::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:39:18,839::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:39:18,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:39:18,841::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:39:18,842::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:39:18,843::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:39:18,843::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:39:18,843::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:39:18,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:19,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:19,295::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:39:19,296::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:39:19,298::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:39:19,300::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:39:19,300::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:39:19,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:19,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:19,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:19,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00614015 s, 667 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:39:19,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:39:19,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168419 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:29,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:29,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00236284 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:39:29,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:39:29,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157995 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:39,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:39,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151446 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:39:39,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:39:39,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730179 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:39:49,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:39:49,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899257 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:39:49,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:39:50,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844603 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14078::WARNING::2013-03-17 10:39:57,774::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:39:57,778::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 10:39:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:40:00,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086377 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:40:00,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:40:00,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093911 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:40:07,781::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:40:07,782::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:40:08,218::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:40:08,225::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:40:08,226::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:40:10,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:40:10,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:40:10,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104355 s, 393 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:40:10,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146367 s, 280 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:40:20,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:40:20,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737527 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:40:20,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:40:20,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904831 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:40:30,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:40:30,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921539 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:40:30,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:40:30,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935002 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:40:40,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:40:40,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832344 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:40:40,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:40:40,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953732 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:40:50,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:40:50,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.029034 s, 141 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:40:50,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:40:50,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103924 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:41:00,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:41:00,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145397 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:41:00,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:41:00,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165316 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14079::WARNING::2013-03-17 10:41:08,254::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:41:08,258::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:41:10,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:41:10,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822314 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:41:10,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:41:10,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935291 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:41:18,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:41:18,262::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:41:18,692::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:41:18,699::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:41:18,700::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:41:20,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:41:20,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174337 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:41:20,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:41:20,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807739 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:41:30,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:41:30,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158228 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:41:30,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:41:30,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157087 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:41:40,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:41:40,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134721 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:41:40,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:41:40,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918727 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:41:50,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:41:50,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00850716 s, 481 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:41:50,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:41:50,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802484 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:42:00,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:42:00,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131031 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:42:01,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:42:01,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807385 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:42:10,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:42:10,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153239 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:42:11,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:42:11,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014008 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14080::WARNING::2013-03-17 10:42:18,740::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:42:18,743::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:42:20,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:42:20,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231502 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:42:21,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:42:21,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085206 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:42:28,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:42:28,746::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:42:29,157::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:42:29,164::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:42:29,165::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:42:30,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:42:30,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168195 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:42:31,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:42:31,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096809 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:42:40,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:42:40,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814574 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:42:41,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:42:41,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163496 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:42:50,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:42:50,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858806 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:42:51,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:42:51,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834581 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:00,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:01,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125373 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:43:01,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:01,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154898 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:11,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:11,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164098 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:43:11,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:11,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749896 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:21,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:21,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156414 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:43:21,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:21,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016154 s, 254 kB/s\n'; <rc> = 0 >Thread-14081::WARNING::2013-03-17 10:43:29,192::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:43:29,195::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:43:31,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:31,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864276 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:43:31,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:31,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974001 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:43:39,197::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:43:39,198::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:43:39,199::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:43:39,624::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:43:39,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:43:39,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:41,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:41,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112341 s, 365 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:43:41,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:41,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854619 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:51,201::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:43:51,203::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:43:51,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:51,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000725049 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:51,231::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000725049 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:43:51,231::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:43:51,231::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:43:51,233::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:43:51,234::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:51,235::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:51,235::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:51,236::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:51,236::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:51,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:51,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:51,654::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:43:51,656::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:43:51,659::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:43:51,660::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:43:51,661::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:43:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:51,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:43:51,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014896 s, 275 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:52,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:43:52,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:43:52,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182609 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:01,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:01,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123094 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:44:02,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:02,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114382 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:12,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:12,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172203 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:44:12,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:12,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302227 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:22,117::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:44:22,118::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:44:22,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:22,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145146 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:22,162::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145146 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:44:22,162::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:44:22,163::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:44:22,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:44:22,165::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:44:22,166::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:44:22,166::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:44:22,167::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:44:22,167::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:44:22,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:22,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:22,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153787 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:22,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:22,609::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:44:22,610::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:44:22,612::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:44:22,613::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:44:22,614::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:44:22,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:22,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:22,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:23,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713873 s, 574 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:44:32,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:32,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140027 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:33,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:33,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0006771 s, 6.0 MB/s\n'; <rc> = 0 >Thread-14082::WARNING::2013-03-17 10:44:39,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:44:39,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:44:42,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:42,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805789 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:43,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:43,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105267 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:44:49,671::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:44:49,672::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:44:50,076::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:44:50,083::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:44:50,084::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:44:52,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:44:52,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150587 s, 272 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:44:53,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:44:53,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082007 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:45:02,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:45:02,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824523 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:45:03,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:45:03,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781177 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:45:12,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:45:12,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161018 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:45:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:45:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830928 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:45:22,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:45:22,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149949 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:45:23,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:45:23,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140533 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:45:32,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:45:32,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140788 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:45:33,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:45:33,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156786 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:45:42,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:45:42,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162367 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:45:43,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:45:43,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0068279 s, 600 kB/s\n'; <rc> = 0 >Thread-14083::WARNING::2013-03-17 10:45:50,109::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:45:50,113::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:45:52,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:45:52,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144385 s, 284 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:45:53,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:45:53,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788082 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:46:00,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:46:00,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:46:00,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:46:00,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:46:00,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:46:02,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:46:02,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111032 s, 369 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:46:03,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:46:03,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851323 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:46:12,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:46:12,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197314 s, 208 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:46:13,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:46:13,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124147 s, 330 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:46:22,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:46:22,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140688 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:46:23,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:46:23,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152019 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:46:32,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:46:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806939 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:46:33,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:46:34,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143894 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:46:42,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:46:42,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799504 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:46:44,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:46:44,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00923094 s, 444 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:46:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:46:52,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156232 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:46:54,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:46:54,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157783 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14084::WARNING::2013-03-17 10:47:00,620::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:47:00,623::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:47:02,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:47:03,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114469 s, 358 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:47:04,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:47:04,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106949 s, 3.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:47:10,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:47:10,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:47:11,063::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:47:11,070::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:47:11,071::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:47:13,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:47:13,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146395 s, 280 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:47:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:47:14,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814553 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:47:23,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:47:23,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0255559 s, 160 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:47:24,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:47:24,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017073 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:47:33,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:47:33,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000682297 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:47:34,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:47:34,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930291 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:47:43,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:47:43,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768829 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:47:44,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:47:44,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151221 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:47:53,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:47:53,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774318 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:47:54,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:47:54,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144027 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:03,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:03,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173269 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:48:04,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:48:04,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144138 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14085::WARNING::2013-03-17 10:48:11,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:48:11,104::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:48:13,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:13,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763078 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:48:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:48:14,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100319 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:48:21,111::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:48:21,112::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:48:21,516::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:48:21,523::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:48:21,524::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:23,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:23,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183553 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:48:24,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:48:24,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794163 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:33,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:33,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145099 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:48:34,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:48:34,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798754 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:43,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:43,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147405 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:48:44,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:48:44,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110258 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:53,529::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:48:53,530::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:48:53,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:53,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00146031 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:53,607::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00146031 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:48:53,607::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:48:53,608::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:48:53,609::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:48:53,610::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:53,611::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:53,611::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:53,612::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:53,612::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:53,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:54,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:54,051::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:48:54,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:48:54,054::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:48:54,055::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:48:54,056::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:48:54,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:54,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:48:54,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:48:54,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151707 s, 270 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:48:55,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:48:55,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149883 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:49:04,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:49:04,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104169 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:05,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:05,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085235 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:49:14,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:49:14,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873619 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:15,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:15,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764798 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14086::WARNING::2013-03-17 10:49:21,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:49:21,561::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:49:24,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:49:24,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976643 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:25,266::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:49:25,268::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:49:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000758553 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:25,295::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000758553 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:49:25,296::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:49:25,296::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:49:25,298::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:49:25,299::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:49:25,300::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:49:25,300::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:49:25,300::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:49:25,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:49:25,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:25,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:25,736::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:49:25,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:49:25,740::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:49:25,741::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:49:25,742::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:49:25,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:26,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:26,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:26,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00538628 s, 760 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:49:31,563::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:49:31,564::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:49:31,565::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:49:31,977::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:49:31,984::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:49:31,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:49:34,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:49:34,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147697 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:36,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:36,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167026 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:49:44,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:49:44,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771631 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:46,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:46,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834454 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:49:54,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:49:54,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943117 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:49:56,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:49:56,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861973 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:50:04,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:50:04,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013542 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:50:06,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:50:06,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151543 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:50:14,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:50:14,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140222 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:50:16,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:50:16,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847288 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:50:24,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:50:24,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156527 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:50:26,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:50:26,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314705 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14087::WARNING::2013-03-17 10:50:32,013::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:50:32,017::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:50:34,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:50:34,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149548 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:50:36,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:50:36,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957362 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:50:42,019::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:50:42,020::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:50:42,515::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:50:42,521::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:50:42,523::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:50:44,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:50:44,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010329 s, 397 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:50:46,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:50:46,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800648 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:50:54,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:50:55,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224413 s, 183 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:50:56,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:50:56,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086747 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:51:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:51:05,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137124 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:51:06,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:51:06,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169607 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:51:15,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:51:15,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735775 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:51:16,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:51:16,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140699 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:51:25,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:51:25,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284543 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:51:26,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:51:27,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169275 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:51:35,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:51:35,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134208 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:51:37,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:51:37,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150708 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14088::WARNING::2013-03-17 10:51:42,556::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:51:42,562::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:51:45,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:51:45,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939393 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:51:47,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:51:47,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946865 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:51:52,564::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:51:52,565::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:51:53,022::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:51:53,029::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:51:53,030::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:51:55,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:51:55,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123955 s, 330 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:51:57,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:51:57,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137538 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:52:05,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:52:05,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898638 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:52:07,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:52:07,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136717 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:52:15,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:52:15,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000696779 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:52:17,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:52:17,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906959 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:52:25,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:52:25,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147835 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:52:27,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:52:27,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107161 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:52:35,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:52:35,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142013 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:52:37,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:52:37,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139107 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:52:45,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:52:45,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152544 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:52:47,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:52:47,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763156 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14089::WARNING::2013-03-17 10:52:53,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:52:53,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:52:55,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:52:55,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00320968 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:52:57,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:52:57,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079674 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:53:03,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:53:03,060::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:53:03,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:53:03,489::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:53:03,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:05,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:05,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165967 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:53:07,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:53:07,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870572 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:15,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:15,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088725 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:53:17,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:53:17,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991768 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:25,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:25,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996609 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:53:27,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:53:27,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935706 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:35,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:35,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128484 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:53:37,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:53:37,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822569 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:45,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:45,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154696 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:53:48,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:53:48,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858361 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:55,763::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:53:55,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:53:55,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:55,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00802862 s, 255 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:55,808::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00802862 s, 255 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:53:55,808::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:53:55,809::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:53:55,810::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:53:55,811::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:55,812::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:55,812::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:55,812::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:55,813::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:55,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:56,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:56,241::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:53:56,243::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:53:56,245::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:53:56,246::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:53:56,247::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:53:56,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:56,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:53:56,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:53:56,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137853 s, 297 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:53:58,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:53:58,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165972 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14090::WARNING::2013-03-17 10:54:03,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:54:03,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:54:06,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:54:06,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199142 s, 206 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:08,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:08,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797116 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:54:13,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:54:13,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:54:14,064::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:54:14,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:54:14,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:54:16,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:54:16,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111721 s, 367 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:18,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:18,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798117 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:54:26,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:54:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153877 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:28,301::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:54:28,302::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:54:28,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:28,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000818898 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:28,331::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000818898 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:54:28,331::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:54:28,332::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:54:28,334::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:54:28,335::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:54:28,335::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:54:28,336::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:54:28,336::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:54:28,336::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:54:28,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:28,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:28,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:54:28,753::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:54:28,755::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:54:28,756::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:54:28,757::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:54:28,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:29,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:29,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:29,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130121 s, 315 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:54:36,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:54:36,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937057 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:39,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:39,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936805 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:54:46,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:54:46,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813649 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:49,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:49,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881607 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:54:56,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:54:56,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987418 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:54:59,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:54:59,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079092 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:55:06,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:55:06,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197957 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:55:09,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:55:09,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135573 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14091::WARNING::2013-03-17 10:55:14,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:55:14,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:55:16,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:55:17,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777297 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:55:19,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:55:19,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854651 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:55:24,101::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 10:55:24,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:55:24,103::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:55:24,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:55:24,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:55:24,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:55:27,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:55:27,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101184 s, 405 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:55:29,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:55:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116605 s, 351 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:55:37,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:55:37,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084364 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:55:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:55:39,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014356 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:55:47,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:55:47,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150208 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:55:49,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:55:49,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150302 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:55:57,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:55:57,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859322 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:55:59,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:55:59,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775448 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:56:07,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:56:07,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161637 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:56:09,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:56:09,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165913 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:56:17,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:56:17,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127479 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:56:19,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:56:19,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157129 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14092::WARNING::2013-03-17 10:56:24,538::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:56:24,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:56:27,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:56:27,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973847 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:56:30,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:56:30,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920834 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:56:34,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:56:34,544::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:56:34,942::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:56:34,948::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:56:34,950::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:56:37,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:56:37,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112076 s, 365 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:56:40,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:56:40,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844956 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:56:47,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:56:47,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860278 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:56:50,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:56:50,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840183 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:56:57,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:56:57,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841483 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:57:00,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:57:00,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819381 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:57:07,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:57:07,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150194 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:57:10,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:57:10,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930332 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:57:17,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:57:17,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015521 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:57:20,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:57:20,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745329 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:57:27,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:57:27,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786573 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:57:30,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:57:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170948 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14093::WARNING::2013-03-17 10:57:34,981::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:57:34,984::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:57:37,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:57:37,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843602 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:57:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:57:40,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848712 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:57:44,986::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:57:44,988::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:57:45,420::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:57:45,426::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:57:45,428::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:57:47,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:57:47,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160023 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:57:50,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:57:50,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113692 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:57:57,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:57:57,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824066 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:58:00,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:58:00,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720594 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:07,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:07,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705527 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:58:10,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:58:10,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961607 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:17,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:17,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829946 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:58:20,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:58:20,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202612 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:27,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:27,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00241444 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:58:30,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:58:30,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147567 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:37,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:37,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183795 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:58:40,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:58:41,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147403 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14094::WARNING::2013-03-17 10:58:45,454::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:58:45,457::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:58:47,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:47,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104222 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:58:51,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:58:51,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812367 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 10:58:55,459::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 10:58:55,460::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 10:58:55,913::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 10:58:55,920::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 10:58:55,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:57,878::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 10:58:57,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:58:57,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:57,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0152509 s, 134 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:57,936::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0152509 s, 134 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 10:58:57,937::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 10:58:57,937::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 10:58:57,939::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 10:58:57,940::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:57,940::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:57,941::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:57,941::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:57,941::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:57,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:58,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:58,340::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 10:58:58,342::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 10:58:58,344::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 10:58:58,345::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 10:58:58,346::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 10:58:58,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:58,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:58:58,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:58:58,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178202 s, 230 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:01,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:01,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078948 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:59:08,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:59:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879694 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:11,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:11,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263856 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:59:18,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:59:18,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082093 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:21,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:21,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016895 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:59:28,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:59:28,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908543 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:31,374::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 10:59:31,376::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:59:31,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:31,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152639 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:31,422::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152639 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 10:59:31,423::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 10:59:31,423::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 10:59:31,424::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 10:59:31,425::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:59:31,426::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:59:31,426::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:59:31,427::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:59:31,427::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 10:59:31,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:31,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:31,859::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 10:59:31,860::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 10:59:31,863::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 10:59:31,864::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 10:59:31,865::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 10:59:31,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:32,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:32,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00368849 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:59:38,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:59:38,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861252 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:42,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:42,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153438 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 10:59:48,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:59:48,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817581 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 10:59:52,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 10:59:52,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817375 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14095::WARNING::2013-03-17 10:59:55,948::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 10:59:55,955::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 10:59:58,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 10:59:58,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808092 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:00:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:00:02,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866684 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:00:05,957::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:00:05,958::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:00:06,409::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:00:06,416::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:00:06,417::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:00:08,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:00:09,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140958 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:00:12,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:00:12,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927731 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:00:19,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:00:19,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100819 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:00:22,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:00:22,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835382 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:00:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:00:29,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154552 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:00:32,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:00:32,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775139 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:00:39,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:00:39,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760707 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:00:42,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:00:42,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170364 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:00:49,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:00:49,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170832 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:00:52,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:00:52,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144953 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:00:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:00:59,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156015 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:01:02,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:01:02,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139393 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14096::WARNING::2013-03-17 11:01:06,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:01:06,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:01:09,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:01:09,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140577 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:01:12,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:01:13,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936887 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:01:16,450::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:01:16,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:01:16,452::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:01:16,931::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:01:16,940::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:01:16,942::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:01:19,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:01:19,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154429 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:01:23,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:01:23,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087855 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:01:29,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:01:29,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134245 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:01:33,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:01:33,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821995 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:01:39,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:01:39,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161011 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:01:43,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:01:43,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141355 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:01:49,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:01:49,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145828 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:01:53,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:01:53,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986067 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:01:59,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:01:59,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837774 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:02:03,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:02:03,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933318 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:02:09,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:02:09,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185262 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:02:13,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:02:13,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744886 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14097::WARNING::2013-03-17 11:02:16,975::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:02:16,980::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:02:19,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:02:19,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253533 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:02:23,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:02:23,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845521 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:02:26,982::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:02:26,983::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:02:27,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:02:27,424::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:02:27,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:02:29,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:02:29,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104048 s, 394 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:02:33,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:02:33,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779621 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:02:39,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:02:39,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921754 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:02:43,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:02:43,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924581 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:02:49,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:02:49,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162975 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:02:53,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:02:53,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888802 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:02:59,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:02:59,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887844 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:03:03,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:03:03,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141811 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:03:09,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:03:09,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00672428 s, 609 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:03:13,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:03:13,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149722 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:03:19,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:03:19,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168401 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:03:23,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:03:24,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139371 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14098::WARNING::2013-03-17 11:03:27,451::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:03:27,454::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:03:29,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:03:29,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904594 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:03:34,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:03:34,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822121 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:03:37,456::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:03:37,457::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:03:37,910::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:03:37,916::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:03:37,918::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:03:39,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:03:39,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154259 s, 266 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:03:44,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:03:44,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888295 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:03:49,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:03:49,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086939 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:03:54,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:03:54,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798443 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:03:59,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:03:59,982::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:03:59,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:00,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00151783 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:04:00,024::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00151783 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:04:00,024::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:04:00,025::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:04:00,026::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:04:00,027::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:00,028::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:00,028::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:00,029::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:00,029::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:00,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:00,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:04:00,474::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:00,475::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:04:00,477::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:04:00,479::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:04:00,479::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:04:00,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:00,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:04:00,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:00,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232679 s, 176 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:04,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:04,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850152 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:04:10,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:11,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213752 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:14,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:14,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147497 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:04:21,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:21,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140936 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:24,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:24,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155862 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:04:31,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:31,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016752 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:34,497::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:04:34,498::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:04:34,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:34,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000809069 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:34,530::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000809069 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:04:34,530::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:04:34,531::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:04:34,533::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:04:34,534::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:04:34,535::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:04:34,536::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:04:34,537::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:04:34,537::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:04:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:34,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:34,979::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:04:34,980::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:04:34,983::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:04:34,984::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:04:34,984::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:04:34,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:35,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:35,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:35,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00606909 s, 675 kB/s\n'; <rc> = 0 >Thread-14099::WARNING::2013-03-17 11:04:37,943::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:04:37,947::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:04:41,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:41,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299682 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:45,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:45,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893029 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:04:47,949::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:04:47,950::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:04:48,391::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:04:48,399::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:04:48,402::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:04:51,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:04:51,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125258 s, 327 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:04:55,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:04:55,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921848 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:05:01,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:05:01,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0367641 s, 111 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:05:05,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:05:05,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176429 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:05:11,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:05:11,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00773652 s, 529 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:05:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:05:15,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270436 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:05:21,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:05:21,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924685 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:05:25,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:05:25,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765304 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:05:31,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:05:31,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131555 s, 311 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:05:35,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:05:35,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015063 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:05:41,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:05:41,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169142 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:05:45,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:05:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986925 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14100::WARNING::2013-03-17 11:05:48,428::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:05:48,431::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:05:51,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:05:51,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171999 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:05:56,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:05:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879188 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:05:58,434::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:05:58,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:05:58,860::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:05:58,867::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:05:58,868::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:06:01,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:06:01,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119328 s, 343 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:06:06,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:06:06,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896232 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:06:11,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:06:11,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088265 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:06:16,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:06:16,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785769 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:06:21,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:06:21,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100743 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:06:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:06:26,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865625 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:06:31,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:06:31,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135618 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:06:36,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:06:36,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147471 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:06:41,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:06:41,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812242 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:06:46,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:06:46,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157452 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:06:51,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:06:51,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141061 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:06:56,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:06:56,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231924 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14101::WARNING::2013-03-17 11:06:58,896::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:06:58,899::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:07:01,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:07:01,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082851 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:07:06,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:07:06,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764925 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:07:08,904::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:07:08,905::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:07:08,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:07:09,352::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:07:09,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:07:09,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:07:11,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:07:11,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128024 s, 320 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:07:16,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:07:16,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152289 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:07:21,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:07:21,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737522 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:07:26,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:07:26,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892096 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:07:31,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:07:31,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000597711 s, 6.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:07:36,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:07:36,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131058 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:07:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:07:41,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195479 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:07:46,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:07:46,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100457 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:07:52,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:07:52,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737288 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:07:56,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:07:56,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077489 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:08:02,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:08:02,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00491601 s, 833 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:08:07,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:08:07,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145667 s, 281 kB/s\n'; <rc> = 0 >Thread-14102::WARNING::2013-03-17 11:08:09,394::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:08:09,400::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:08:12,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:08:12,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879388 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:08:17,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:08:17,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087487 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:08:19,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:08:19,405::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:08:19,867::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:08:19,874::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:08:19,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:08:22,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:08:22,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164032 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:08:27,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:08:27,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117083 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:08:32,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:08:32,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115648 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:08:37,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:08:37,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963077 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:08:42,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:08:42,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826217 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:08:47,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:08:47,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185131 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:08:52,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:08:52,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144527 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:08:57,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:08:57,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134807 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:02,309::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:09:02,310::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:09:02,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:02,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128842 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:02,349::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128842 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:09:02,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:09:02,350::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:09:02,353::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:09:02,354::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:02,354::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:02,355::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:02,355::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:02,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:02,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:02,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:02,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:02,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:09:02,806::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:09:02,807::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:09:02,807::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:09:02,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:03,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:03,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:03,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140982 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:07,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:07,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150646 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:13,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:13,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801333 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:17,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:17,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138617 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14103::WARNING::2013-03-17 11:09:19,905::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:09:19,909::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:09:23,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:23,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817833 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:27,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:27,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931027 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:09:29,911::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:09:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:09:30,377::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:09:30,384::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:09:30,386::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:09:33,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:33,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147843 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:37,883::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:09:37,885::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:09:37,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:37,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000820267 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:37,913::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000820267 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:09:37,913::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:09:37,914::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:09:37,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:09:37,916::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:09:37,917::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:09:37,917::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:09:37,918::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:09:37,918::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:09:37,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:38,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:38,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:09:38,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:09:38,353::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:09:38,354::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:09:38,355::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:09:38,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:38,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:38,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00679971 s, 602 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:43,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:43,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175303 s, 234 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:48,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:48,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021793 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:09:53,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:09:53,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151724 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:09:58,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:09:58,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00942625 s, 435 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:10:03,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:10:03,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878495 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:10:08,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:10:09,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718011 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:10:13,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:10:13,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829329 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:10:19,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:10:19,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877328 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:10:23,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:10:23,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805968 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:10:29,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:10:29,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00539254 s, 760 kB/s\n'; <rc> = 0 >Thread-14104::WARNING::2013-03-17 11:10:30,424::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:10:30,429::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:10:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:10:33,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996499 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:10:39,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:10:39,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161989 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:10:40,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:10:40,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:10:40,881::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:10:40,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:10:40,889::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:10:43,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:10:43,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156989 s, 261 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:10:49,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:10:49,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772167 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:10:53,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:10:53,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130148 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:10:59,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:10:59,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141307 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:11:03,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:11:03,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795739 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:11:09,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:11:09,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070935 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:11:13,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:11:13,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207519 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:11:19,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:11:19,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126942 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:11:23,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:11:23,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821388 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:11:29,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:11:29,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144902 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:11:33,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:11:33,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163442 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:11:39,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:11:40,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122891 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14105::WARNING::2013-03-17 11:11:40,918::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:11:40,928::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:11:43,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:11:43,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117228 s, 349 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:11:50,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:11:50,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766981 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:11:50,932::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:11:50,934::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:11:51,383::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:11:51,389::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:11:51,390::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:11:53,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:11:53,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132679 s, 309 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:12:00,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:12:00,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813184 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:12:04,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:12:04,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196767 s, 208 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:12:10,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:12:10,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146668 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:12:14,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:12:14,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793182 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:12:20,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:12:20,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858781 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:12:24,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:12:24,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158515 s, 258 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:12:30,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:12:30,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158558 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:12:34,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:12:34,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160785 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:12:40,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:12:40,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265393 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:12:44,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:12:44,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129118 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:12:50,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:12:50,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737736 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14106::WARNING::2013-03-17 11:12:51,415::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:12:51,419::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:12:54,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:12:54,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864705 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:13:01,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:13:01,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161757 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:13:01,424::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:13:01,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:13:01,427::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:13:01,918::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:13:01,924::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:13:01,926::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:13:04,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:13:04,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119569 s, 343 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:13:11,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:13:11,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138004 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:13:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:13:14,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735921 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:13:21,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:13:21,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152253 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:13:24,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:13:24,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121773 s, 336 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:13:31,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:13:31,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873388 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:13:34,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:13:34,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796521 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:13:41,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:13:41,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846912 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:13:44,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:13:44,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012956 s, 316 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:13:51,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:13:51,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884749 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:13:54,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:13:54,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146892 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:01,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:01,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00705599 s, 580 kB/s\n'; <rc> = 0 >Thread-14107::WARNING::2013-03-17 11:14:02,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:14:02,027::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:14:04,550::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:14:04,552::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:14:04,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:04,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133929 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:04,600::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133929 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:14:04,601::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:14:04,601::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:14:04,603::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:14:04,604::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:04,605::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:04,606::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:04,606::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:04,606::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:04,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:05,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:05,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:05,070::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:14:05,073::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:14:05,074::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:14:05,075::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:14:05,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:05,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:05,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:05,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0448401 s, 91.3 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:12,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 11:14:12,045::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:14:12,076::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:12,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00909387 s, 450 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:14:12,523::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:14:12,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:14:12,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:14:15,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:15,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186032 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:22,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:22,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914831 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:25,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:25,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747364 s, 548 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:32,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:32,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771268 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:35,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:35,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911849 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:42,286::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:14:42,288::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:14:42,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:42,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015973 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:42,335::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015973 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:14:42,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:14:42,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:14:42,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:14:42,339::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:14:42,339::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:14:42,340::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:14:42,340::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:14:42,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:14:42,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:42,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:42,780::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:14:42,781::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:14:42,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:14:42,785::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:14:42,785::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:14:42,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:43,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:43,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:43,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00815123 s, 503 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:45,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:45,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.046092 s, 88.9 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:14:53,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:14:53,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015009 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:14:55,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:14:55,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737941 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:15:03,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:15:03,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748701 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:15:05,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:15:05,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848364 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14108::WARNING::2013-03-17 11:15:12,552::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:15:12,556::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:15:13,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:15:13,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798234 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:15:15,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:15:15,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932625 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:15:22,558::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:15:22,559::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:15:22,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:15:22,985::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:15:22,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:15:23,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:15:23,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146047 s, 280 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:15:25,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:15:25,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103289 s, 397 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:15:33,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:15:33,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00999994 s, 410 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:15:35,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:15:35,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139812 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:15:43,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:15:43,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150259 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:15:45,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:15:46,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908362 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:15:53,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:15:53,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164833 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:15:56,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:15:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139736 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:16:03,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:16:03,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169894 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:16:06,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:16:06,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780142 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:16:13,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:16:13,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104963 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:16:16,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:16:16,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175046 s, 234 kB/s\n'; <rc> = 0 >Thread-14109::WARNING::2013-03-17 11:16:23,015::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:16:23,020::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:16:23,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:16:24,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772014 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:16:26,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:16:26,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844657 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:16:33,024::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:16:33,025::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:16:33,427::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:16:33,434::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:16:33,435::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:16:34,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:16:34,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159883 s, 256 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:16:36,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:16:36,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121929 s, 336 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:16:44,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:16:44,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110012 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:16:46,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:16:46,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244396 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:16:54,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:16:54,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092722 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:16:56,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:16:56,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000702183 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:17:04,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:17:04,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174316 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:17:06,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:17:06,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145803 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:17:14,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:17:14,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759751 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:17:16,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:17:16,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778376 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:17:24,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:17:24,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152858 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:17:26,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:17:26,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829499 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14110::WARNING::2013-03-17 11:17:33,465::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:17:33,468::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:17:34,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:17:34,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855909 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:17:36,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:17:36,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785941 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:17:43,470::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:17:43,471::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:17:43,914::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:17:43,921::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:17:43,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:17:44,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:17:44,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136368 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:17:46,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:17:46,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116951 s, 350 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:17:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:17:54,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860526 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:17:56,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:17:56,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832773 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:18:04,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:18:04,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164422 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:18:06,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:18:06,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903113 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:18:14,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:18:14,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748253 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:18:16,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:18:16,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148121 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:18:24,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:18:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135105 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:18:26,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:18:26,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0468389 s, 87.4 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:18:34,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:18:35,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177524 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:18:36,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:18:36,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124914 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14111::WARNING::2013-03-17 11:18:43,958::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:18:43,964::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:18:45,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:18:45,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813654 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:18:46,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:18:46,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934645 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:18:53,966::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:18:53,967::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:18:53,968::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:18:54,390::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:18:54,396::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:18:54,398::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:18:55,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:18:55,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123796 s, 331 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:18:56,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:18:57,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163488 s, 251 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:05,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:05,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812861 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:07,005::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:19:07,013::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:19:07,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:07,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135206 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:07,058::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135206 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:19:07,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:19:07,058::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:19:07,060::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:19:07,061::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:19:07,061::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:19:07,062::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:19:07,062::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:19:07,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:19:07,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:07,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:07,482::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:19:07,483::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:19:07,485::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:19:07,487::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:19:07,488::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:19:07,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:07,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:07,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:07,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114611 s, 357 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:15,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:15,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820352 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:17,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:18,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152616 s, 268 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:25,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899197 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:28,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:28,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185841 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:35,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:35,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774841 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:38,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:38,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140323 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:45,501::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:19:45,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:19:45,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:45,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000934773 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:45,534::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000934773 s, 2.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:19:45,535::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:19:45,535::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:19:45,537::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:19:45,538::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:19:45,539::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:19:45,539::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:19:45,539::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:19:45,540::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:19:45,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:45,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:45,992::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:19:45,993::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:19:45,995::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:19:45,996::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:19:45,997::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:19:45,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:19:46,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:46,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00407152 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:48,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:48,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137336 s, 298 kB/s\n'; <rc> = 0 >Thread-14112::WARNING::2013-03-17 11:19:54,424::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:19:54,427::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:19:56,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:19:56,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878313 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:19:58,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:19:58,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879318 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:20:04,430::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:20:04,431::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:20:04,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:20:04,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:20:04,865::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:20:06,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:20:06,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131317 s, 312 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:20:08,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:20:08,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184611 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:20:16,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:20:16,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157328 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:20:18,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:20:18,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016573 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:20:26,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:20:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741557 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:20:28,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:20:28,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016317 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:20:36,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:20:36,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150635 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:20:38,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:20:38,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281477 s, 146 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:20:46,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:20:46,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167843 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:20:48,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:20:48,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137014 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:20:56,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:20:56,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143416 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:20:58,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:20:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152678 s, 268 kB/s\n'; <rc> = 0 >Thread-14113::WARNING::2013-03-17 11:21:04,892::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:21:04,896::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:21:07,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:21:07,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771811 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:21:08,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:21:08,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830814 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:21:14,898::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:21:14,899::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:21:15,421::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:21:15,428::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:21:15,430::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:21:17,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:21:17,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162056 s, 253 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:21:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:21:18,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151962 s, 270 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:21:27,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:21:27,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795026 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:21:28,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:21:28,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966688 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:21:37,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:21:37,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150189 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:21:38,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:21:38,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00585461 s, 700 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:21:47,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:21:47,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875708 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:21:48,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:21:48,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143132 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:21:57,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:21:57,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148195 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:21:58,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:21:58,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114306 s, 358 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:22:07,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:22:07,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145739 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:22:08,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:22:08,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290717 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14114::WARNING::2013-03-17 11:22:15,487::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:22:15,492::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:22:17,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:22:17,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186522 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:22:18,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:22:18,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00492444 s, 832 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:22:25,494::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:22:25,495::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:22:26,140::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:22:26,147::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:22:26,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:22:27,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:22:27,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812283 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:22:28,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:22:28,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133934 s, 306 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:22:37,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:22:37,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00468752 s, 874 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:22:38,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:22:38,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011401 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:22:47,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:22:47,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826771 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:22:48,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:22:48,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143131 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:22:57,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:22:57,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135358 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:22:58,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:22:59,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114242 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:23:07,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:23:07,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144326 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:23:09,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:23:09,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748419 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:23:18,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:23:18,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748405 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:23:19,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:23:19,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144529 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14115::WARNING::2013-03-17 11:23:26,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:23:26,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:23:28,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:23:28,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140069 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:23:29,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:23:29,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796299 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:23:36,185::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:23:36,186::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:23:36,662::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:23:36,668::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:23:36,670::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:23:38,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:23:38,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137079 s, 299 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:23:39,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:23:39,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164387 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:23:48,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:23:48,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802128 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:23:49,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:23:49,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150956 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:23:58,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:23:58,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103578 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:23:59,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:23:59,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880132 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:08,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:08,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101857 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:09,297::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:24:09,299::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:24:09,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:09,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0011776 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:09,341::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0011776 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:24:09,342::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:24:09,342::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:24:09,344::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:24:09,345::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:24:09,345::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:24:09,346::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:24:09,346::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:24:09,347::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:24:09,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:09,783::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:24:09,783::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:24:09,786::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:24:09,787::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:24:09,788::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:24:09,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:10,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:10,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:10,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759538 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:18,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:18,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923762 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:20,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:20,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136989 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:28,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:28,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156685 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:30,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:30,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015565 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14116::WARNING::2013-03-17 11:24:36,705::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:24:36,711::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:24:38,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:38,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019112 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:40,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:40,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787987 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:24:46,714::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:24:46,715::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:24:46,717::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:24:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:24:47,193::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:24:47,195::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:48,742::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:24:48,745::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:24:48,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:48,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140945 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:48,785::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140945 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:24:48,785::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:24:48,785::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:24:48,787::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:24:48,788::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:48,789::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:48,789::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:48,790::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:48,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:48,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:49,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:49,158::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:24:49,159::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:24:49,162::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:24:49,163::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:24:49,163::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:24:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:49,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:49,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:49,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109516 s, 374 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:24:50,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:24:50,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00606991 s, 675 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:24:59,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:24:59,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845788 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:25:00,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:25:00,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896748 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:25:09,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:25:09,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115167 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:25:10,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:25:10,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141203 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:25:19,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:25:19,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155782 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:25:20,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:25:20,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126736 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:25:29,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:25:29,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172427 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:25:30,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:25:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864563 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:25:39,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:25:39,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146516 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:25:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:25:40,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146377 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14117::WARNING::2013-03-17 11:25:47,222::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:25:47,226::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:25:50,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:25:50,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811351 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:25:50,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:25:50,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895372 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:25:57,228::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:25:57,230::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:25:57,711::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:25:57,717::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:25:57,718::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:26:00,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:26:00,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958877 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:26:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:26:00,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127594 s, 321 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:26:10,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:26:10,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169789 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:26:10,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:26:10,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151509 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:26:20,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:26:20,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982891 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:26:20,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:26:20,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107834 s, 380 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:26:30,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:26:30,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792011 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:26:30,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:26:30,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182977 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:26:40,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:26:40,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111752 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:26:40,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:26:40,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086306 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:26:50,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:26:50,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925991 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:26:50,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:26:50,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193579 s, 212 kB/s\n'; <rc> = 0 >Thread-14118::WARNING::2013-03-17 11:26:57,752::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:26:57,755::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:27:00,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:27:00,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993462 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:27:00,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:27:00,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834619 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:27:07,762::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:27:07,763::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:27:08,296::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:27:08,303::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:27:08,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:27:10,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:27:10,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913934 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:27:10,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:27:10,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013527 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:27:20,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:27:20,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843384 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:27:20,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:27:20,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166816 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:27:30,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:27:30,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0031533 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:27:30,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:27:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887059 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:27:40,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:27:40,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115421 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:27:41,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:27:41,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176465 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:27:50,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:27:50,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801405 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:27:51,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:27:51,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156014 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:28:00,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:28:00,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893817 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:28:01,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:28:01,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768526 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14119::WARNING::2013-03-17 11:28:08,328::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:28:08,332::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:28:11,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:28:11,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185639 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:28:11,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:28:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808383 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:28:18,334::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:28:18,335::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:28:18,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:28:18,838::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:28:18,840::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:28:21,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:28:21,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962278 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:28:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:28:21,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135503 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:28:31,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:28:31,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147196 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:28:31,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:28:31,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984387 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:28:41,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:28:41,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:28:41,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960504 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:28:41,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156343 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:28:51,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:28:51,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:28:51,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168608 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:28:51,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135807 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:01,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:01,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915543 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:01,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:01,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014926 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:11,600::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:29:11,602::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:29:11,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:11,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137717 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:11,643::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137717 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:29:11,643::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:29:11,643::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:29:11,645::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:29:11,649::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:29:11,649::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:29:11,650::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:29:11,650::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:29:11,651::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:29:11,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:11,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:11,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148286 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:12,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:12,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:29:12,119::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:29:12,123::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:29:12,124::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:29:12,125::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:29:12,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:12,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:12,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:12,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147638 s, 277 kB/s\n'; <rc> = 0 >Thread-14120::WARNING::2013-03-17 11:29:18,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:29:18,874::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:29:21,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:21,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114601 s, 357 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:22,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:22,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941567 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:29:28,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:29:28,886::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:29:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:29:29,386::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:29:29,388::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:31,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:31,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148613 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:32,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:32,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116513 s, 352 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:41,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:41,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153688 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:29:42,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091336 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:52,026::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:29:52,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:29:52,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:52,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000761892 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:52,056::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000761892 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:29:52,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:29:52,057::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:29:52,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:29:52,060::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:52,060::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:52,061::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:52,061::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:52,062::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:52,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:52,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:52,476::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:29:52,477::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:29:52,479::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:29:52,481::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:29:52,481::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:29:52,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:52,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:29:52,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201059 s, 204 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:29:52,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:29:52,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110695 s, 370 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:30:02,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:30:02,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128297 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:30:02,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:30:02,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160877 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:30:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:30:12,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132421 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:30:13,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:30:13,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163618 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:30:22,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:30:22,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148127 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:30:23,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:30:23,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166582 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14121::WARNING::2013-03-17 11:30:29,413::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:30:29,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:30:32,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:30:32,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809813 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:30:33,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:30:33,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818425 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:30:39,418::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:30:39,419::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:30:39,420::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:30:39,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:30:39,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:30:39,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:30:42,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:30:42,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157535 s, 260 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:30:43,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:30:43,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083269 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:30:52,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:30:53,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968114 s, 423 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:30:53,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:30:53,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014622 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:31:03,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:31:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299402 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:31:03,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:31:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870035 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:31:13,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:31:13,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113085 s, 362 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:31:13,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:31:13,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221965 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:31:23,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:31:23,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152245 s, 269 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:31:23,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:31:23,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763329 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:31:33,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:31:33,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172081 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:31:33,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:31:33,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165727 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14122::WARNING::2013-03-17 11:31:39,863::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:31:39,866::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:31:43,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:31:43,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133343 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:31:43,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:31:43,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00885285 s, 463 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:31:49,881::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:31:49,883::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:31:50,297::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:31:50,304::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:31:50,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:31:53,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:31:53,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00905313 s, 452 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:31:53,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:31:53,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146825 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:32:03,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:32:03,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154196 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:32:03,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:32:03,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081081 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:32:13,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:32:13,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167315 s, 245 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:32:13,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:32:14,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170814 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:32:23,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:32:23,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841363 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:32:24,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:32:24,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118095 s, 347 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:32:33,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:32:33,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136358 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:32:34,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:32:34,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000698876 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:32:43,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:32:43,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00738901 s, 554 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:32:44,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:32:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150695 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14123::WARNING::2013-03-17 11:32:50,332::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:32:50,336::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:32:53,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:32:53,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929796 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:32:54,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:32:54,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019854 s, 2.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:33:00,338::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:33:00,339::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:33:00,779::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:33:00,786::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:33:00,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:33:03,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:33:03,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143679 s, 285 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:33:04,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:33:04,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085115 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:33:13,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:33:13,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800702 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:33:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:33:14,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302352 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:33:23,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:33:23,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133048 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:33:24,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:33:24,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151989 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:33:33,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:33:33,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128782 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:33:34,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:33:34,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158471 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:33:43,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:33:43,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143386 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:33:44,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:33:44,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829116 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:33:53,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:33:53,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769988 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:33:54,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:33:54,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137027 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14124::WARNING::2013-03-17 11:34:00,816::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:34:00,820::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:34:03,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:03,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088959 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:04,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:04,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909511 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:34:10,822::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:34:10,823::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:34:11,266::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:34:11,273::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:34:11,274::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:13,968::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:34:13,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:34:13,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:14,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0158079 s, 130 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:14,053::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0158079 s, 130 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:34:14,053::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:34:14,053::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:34:14,055::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:34:14,056::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:14,056::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:14,057::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:14,057::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:14,057::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:14,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:14,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:14,489::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:34:14,490::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:34:14,493::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:34:14,494::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:34:14,495::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:34:14,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:14,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:14,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:14,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829914 s, 494 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:14,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:14,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148815 s, 275 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:24,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:24,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080001 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:24,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:24,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813625 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:34,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:34,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138688 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:35,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:35,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230397 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:44,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:45,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014269 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:45,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:45,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145567 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:34:55,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:34:55,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821031 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:55,200::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:34:55,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:34:55,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:55,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012458 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:55,241::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012458 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:34:55,242::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:34:55,242::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:34:55,244::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:34:55,245::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:34:55,246::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:34:55,246::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:34:55,246::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:34:55,247::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:34:55,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:55,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:55,695::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:34:55,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:34:55,699::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:34:55,700::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:34:55,700::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:34:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:56,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:34:56,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:34:56,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104599 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:35:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:35:05,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779559 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:35:06,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:35:06,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954568 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14125::WARNING::2013-03-17 11:35:11,301::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:35:11,305::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:35:15,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:35:15,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783458 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:35:16,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:35:16,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846936 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:35:21,307::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:35:21,308::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:35:21,747::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:35:21,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:35:21,758::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:35:25,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:35:25,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0420593 s, 97.4 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:35:26,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:35:26,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931518 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:35:35,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:35:35,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764919 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:35:36,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:35:36,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103026 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:35:45,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:35:45,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137658 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:35:46,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:35:46,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166182 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:35:55,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:35:55,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843507 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:35:56,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:35:56,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837287 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:36:05,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:36:05,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155548 s, 263 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:36:06,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:36:06,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015697 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:36:15,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:36:15,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108109 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:36:16,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:36:16,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191426 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14126::WARNING::2013-03-17 11:36:21,788::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:36:21,793::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:36:25,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:36:25,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102569 s, 399 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:36:26,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:36:26,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000680448 s, 6.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:36:31,795::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:36:31,797::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:36:31,798::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:36:32,193::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:36:32,200::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:36:32,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:36:35,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:36:35,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165569 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:36:36,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:36:36,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194225 s, 211 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:36:45,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:36:45,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188962 s, 217 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:36:46,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:36:47,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001333 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:36:55,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:36:55,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856362 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:36:57,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:36:57,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861598 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:37:05,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:37:05,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212914 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:37:07,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:37:07,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948182 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:37:15,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:37:15,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724319 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:37:17,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:37:17,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838761 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:37:25,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:37:25,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113607 s, 361 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:37:27,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:37:27,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762414 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14127::WARNING::2013-03-17 11:37:32,230::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:37:32,233::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:37:35,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:37:35,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994805 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:37:37,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:37:37,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779501 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:37:42,235::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:37:42,236::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:37:42,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:37:42,682::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:37:42,684::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:37:45,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:37:45,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137793 s, 297 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:37:47,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:37:47,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103525 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:37:55,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:37:55,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008113 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:37:57,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:37:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791568 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:38:05,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:38:05,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070691 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:38:07,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:38:07,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183789 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:38:15,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:38:16,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00386606 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:38:17,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:38:17,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810892 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:38:26,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:38:26,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202516 s, 202 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:38:27,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:38:27,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086945 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:38:36,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:38:36,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993986 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:38:37,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:38:37,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155048 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14128::WARNING::2013-03-17 11:38:42,710::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:38:42,714::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:38:46,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:38:46,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901787 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:38:47,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:38:47,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932875 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:38:52,722::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:38:52,723::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:38:53,138::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:38:53,145::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:38:53,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:38:56,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:38:56,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160765 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:38:57,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:38:57,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758955 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:06,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:06,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926245 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:08,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:08,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180611 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:16,276::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:39:16,277::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:39:16,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:16,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000780198 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:16,306::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000780198 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:39:16,308::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:39:16,311::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:39:16,313::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:39:16,314::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:39:16,314::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:39:16,315::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:39:16,315::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:39:16,316::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:39:16,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:16,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:16,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:39:16,754::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:39:16,757::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:39:16,758::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:39:16,758::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:39:16,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:17,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:17,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:17,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134665 s, 304 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:18,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:18,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083546 s, 490 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:27,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:27,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737556 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:28,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:28,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019715 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:37,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:37,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789307 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:38,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:38,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151153 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:39:47,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:47,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012168 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:48,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:48,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135125 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14129::WARNING::2013-03-17 11:39:53,181::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:39:53,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:39:57,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:39:57,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831693 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:58,410::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:39:58,412::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:39:58,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:58,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000757147 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:58,456::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000757147 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:39:58,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:39:58,457::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:39:58,458::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:39:58,459::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:39:58,460::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:39:58,461::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:39:58,461::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:39:58,461::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:39:58,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:58,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:58,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:39:58,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:39:58,878::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:39:58,879::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:39:58,880::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:39:58,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:59,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:39:59,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:39:59,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00907303 s, 451 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:40:03,189::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:40:03,190::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:40:03,621::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:40:03,627::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:40:03,629::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:40:07,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:40:07,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214625 s, 191 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:40:09,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:40:09,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164122 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:40:17,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:40:17,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145749 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:40:19,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:40:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867073 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:40:27,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:40:27,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148622 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:40:29,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:40:29,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201008 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:40:37,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:40:37,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973445 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:40:39,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:40:39,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134884 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:40:47,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:40:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014143 s, 290 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:40:49,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:40:49,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092458 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:40:57,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:40:57,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842375 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:40:59,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:40:59,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156566 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14130::WARNING::2013-03-17 11:41:03,656::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:41:03,660::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:41:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:41:07,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852875 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:41:09,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:41:09,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857335 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:41:13,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:41:13,669::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:41:14,102::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:41:14,109::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:41:14,110::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:41:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:41:17,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121902 s, 336 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:41:19,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:41:19,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966991 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:41:27,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:41:27,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0368534 s, 111 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:41:29,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:41:30,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788759 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:41:37,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:41:37,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816841 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:41:40,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:41:40,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849494 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:41:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:41:47,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148669 s, 276 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:41:50,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:41:50,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208617 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:41:57,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:41:57,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842871 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:42:00,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:42:00,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885404 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:42:07,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:42:07,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865137 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:42:10,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:42:10,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146415 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14131::WARNING::2013-03-17 11:42:14,137::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:42:14,141::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:42:17,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:42:18,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081246 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:42:20,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:42:20,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161132 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:42:24,142::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:42:24,143::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:42:24,144::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:42:24,602::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:42:24,609::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:42:24,610::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:42:28,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:42:28,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0041052 s, 998 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:42:30,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:42:30,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902742 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:42:38,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:42:38,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993669 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:42:40,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:42:40,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158492 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:42:48,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:42:48,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00365181 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:42:50,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:42:50,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152149 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:42:58,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:42:58,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857315 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:43:00,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:43:00,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226829 s, 181 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:43:08,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:43:08,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808323 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:43:10,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:43:10,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155986 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:43:18,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:43:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880828 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:43:20,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:43:20,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165678 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14132::WARNING::2013-03-17 11:43:24,640::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:43:24,642::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:43:28,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:43:28,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178046 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:43:30,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:43:30,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880815 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:43:34,644::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:43:34,645::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:43:35,073::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:43:35,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:43:35,081::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:43:38,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:43:38,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144388 s, 284 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:43:40,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:43:40,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833432 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:43:48,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:43:48,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113711 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:43:51,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:43:51,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890592 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:43:58,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:43:58,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087261 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:44:01,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:44:01,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149577 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:08,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:08,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165446 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:44:11,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:44:11,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152656 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:18,479::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:44:18,480::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:44:18,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:18,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118141 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:18,521::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118141 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:44:18,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:44:18,522::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:44:18,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:44:18,525::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:18,525::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:18,526::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:18,526::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:18,527::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:18,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:18,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:18,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:18,942::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:44:18,945::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:44:18,946::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:44:18,946::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:44:18,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:19,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:19,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154644 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:44:21,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:44:21,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853302 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:29,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213566 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:44:31,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:44:31,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144984 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14133::WARNING::2013-03-17 11:44:35,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:44:35,119::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:44:39,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:39,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808677 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:44:41,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:44:41,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909018 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:44:45,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:44:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:44:45,578::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:44:45,585::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:44:45,586::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:44:49,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:49,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010943 s, 374 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:44:51,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:44:51,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913181 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:44:59,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:44:59,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820909 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:01,765::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:45:01,766::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:45:01,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:01,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00163562 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:01,813::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00163562 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:45:01,814::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:45:01,814::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:45:01,816::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:45:01,817::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:45:01,817::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:45:01,818::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:45:01,818::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:45:01,818::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:45:01,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:02,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:02,250::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:45:02,251::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:45:02,253::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:45:02,255::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:45:02,255::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:45:02,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:02,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:02,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:02,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141669 s, 289 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:45:09,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:45:09,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193831 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:12,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:12,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785778 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:45:19,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:45:19,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183116 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:22,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:22,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942238 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:45:29,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:45:29,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159697 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:32,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:32,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145302 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:45:39,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:45:39,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162685 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:42,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:43,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716306 s, 5.7 MB/s\n'; <rc> = 0 >Thread-14134::WARNING::2013-03-17 11:45:45,609::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:45:45,613::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:45:49,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:45:49,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817744 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:45:53,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:45:53,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835375 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:45:55,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:45:55,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:45:56,079::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:45:56,086::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:45:56,087::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:45:59,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:45:59,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018303 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:46:03,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:46:03,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010109 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:46:09,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:46:09,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151708 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:46:13,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:46:13,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915716 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:46:19,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:46:19,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999908 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:46:23,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:46:23,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151916 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:46:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:46:29,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766306 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:46:33,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:46:33,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753686 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:46:39,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:46:39,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904833 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:46:43,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:46:43,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174992 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:46:49,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:46:50,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128484 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:46:53,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:46:53,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125043 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14135::WARNING::2013-03-17 11:46:56,112::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:46:56,116::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:47:00,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:47:00,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814761 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:47:03,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:47:03,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185279 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:47:06,119::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:47:06,120::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:47:06,547::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:47:06,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:47:06,554::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:47:10,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:47:10,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183457 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:47:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:47:13,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136811 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:47:20,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:47:20,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831438 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:47:23,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:47:23,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146912 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:47:30,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:47:30,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810698 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:47:33,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:47:33,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158015 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:47:40,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:47:40,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776483 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:47:43,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:47:43,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000682939 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:47:50,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:47:50,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138911 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:47:53,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:47:54,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139786 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:48:00,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:48:00,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950778 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:48:04,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:48:04,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141272 s, 290 kB/s\n'; <rc> = 0 >Thread-14136::WARNING::2013-03-17 11:48:06,585::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:48:06,592::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:48:10,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:48:10,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103035 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:48:14,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:48:14,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733438 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:48:16,604::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:48:16,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:48:16,606::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:48:17,051::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:48:17,057::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:48:17,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:48:20,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:48:20,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174017 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:48:24,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:48:24,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00891909 s, 459 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:48:30,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:48:30,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793381 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:48:34,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:48:34,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857345 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:48:40,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:48:40,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896536 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:48:44,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:48:44,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764143 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:48:50,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:48:50,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824271 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:48:54,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:48:54,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790897 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:00,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:00,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868527 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:49:04,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:49:04,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843654 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:10,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:10,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00236758 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:49:14,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:49:14,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113053 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14137::WARNING::2013-03-17 11:49:17,093::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:49:17,099::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:49:20,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:49:20,596::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:49:20,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:20,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000892112 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:20,626::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000892112 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:49:20,626::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:49:20,626::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:49:20,628::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:49:20,629::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:20,630::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:20,630::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:20,631::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:20,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:20,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:21,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:21,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:21,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:49:21,038::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:49:21,040::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:49:21,040::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:49:21,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:21,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:21,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:21,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01601 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:49:24,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:49:24,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091521 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:49:27,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:49:27,110::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:49:27,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:49:27,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:49:27,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:49:31,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:31,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00689419 s, 594 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:49:34,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:49:34,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010886 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:41,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:41,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805407 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:49:44,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:49:44,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849122 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:49:51,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:49:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143101 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:49:54,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:49:54,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867547 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:50:01,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:50:01,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854955 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:04,951::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:50:04,953::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:50:04,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:05,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129811 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:05,025::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129811 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:50:05,026::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:50:05,026::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:50:05,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:50:05,028::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:50:05,029::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:50:05,030::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:50:05,030::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:50:05,030::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:50:05,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:05,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:05,460::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:50:05,461::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:50:05,465::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:50:05,466::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:50:05,466::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:50:05,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:05,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:05,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:05,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00301715 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:50:11,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:50:11,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140064 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:15,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:15,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145978 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:50:21,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:50:21,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882075 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:25,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:26,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983314 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14138::WARNING::2013-03-17 11:50:27,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:50:27,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:50:31,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:50:31,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863414 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:36,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:36,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258077 s, 1.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:50:37,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:50:37,608::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:50:38,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:50:38,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:50:38,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:50:41,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:50:41,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00493764 s, 830 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:46,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:46,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791265 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:50:51,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:50:51,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146613 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:50:56,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:50:56,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736563 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:51:01,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:51:01,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155308 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:51:06,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:51:06,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112842 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:51:11,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:51:12,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800016 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:51:16,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:51:16,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153423 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:51:22,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:51:22,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163561 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:51:26,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:51:26,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154947 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:51:32,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:51:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144047 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:51:36,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:51:36,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102305 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14139::WARNING::2013-03-17 11:51:38,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:51:38,122::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:51:42,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:51:42,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884673 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:51:46,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:51:46,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838545 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:51:48,124::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:51:48,125::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:51:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:51:48,541::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:51:48,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:51:52,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:51:52,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116324 s, 352 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:51:56,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:51:56,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009024 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:52:02,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:52:02,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845942 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:52:06,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:52:06,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879152 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:52:12,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:52:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014362 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:52:16,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:52:16,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705093 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:52:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:52:22,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00671452 s, 610 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:52:26,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:52:26,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909536 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:52:32,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:52:32,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778347 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:52:37,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:52:37,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013924 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:52:42,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:52:42,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853513 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:52:47,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:52:47,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839959 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14140::WARNING::2013-03-17 11:52:48,570::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:52:48,574::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:52:52,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:52:52,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102086 s, 401 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:52:57,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:52:57,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731664 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:52:58,578::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:52:58,580::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:52:59,005::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:52:59,012::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:52:59,013::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:53:02,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:53:02,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129613 s, 316 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:53:07,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:53:07,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750914 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:53:12,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:53:12,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154712 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:53:17,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:53:17,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737454 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:53:22,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:53:22,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821027 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:53:27,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:53:27,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0419133 s, 97.7 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:53:32,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:53:32,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802909 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:53:37,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:53:37,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012808 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:53:42,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:53:42,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828336 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:53:47,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:53:47,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791279 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:53:52,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:53:52,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106312 s, 385 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:53:57,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:53:58,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016152 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14141::WARNING::2013-03-17 11:53:59,038::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:53:59,042::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 11:54:02,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:02,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155026 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:54:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:54:08,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188619 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:54:09,052::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 11:54:09,053::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:54:09,054::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:54:09,500::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:54:09,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:54:09,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:12,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:12,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165977 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:54:18,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:54:18,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934661 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:22,786::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:54:22,787::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:54:22,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:22,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000845882 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:22,817::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000845882 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:54:22,817::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:54:22,817::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:54:22,819::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:54:22,820::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:22,821::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:22,821::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:22,821::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:22,822::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:22,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:23,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:23,249::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:54:23,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:54:23,252::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:54:23,253::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:54:23,254::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:54:23,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:23,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:23,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:23,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207075 s, 198 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:54:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:54:28,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027488 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:33,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:33,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103218 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:54:38,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:54:38,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730509 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:43,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:43,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171607 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:54:48,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:54:48,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080839 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:54:53,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:54:53,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194402 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:54:58,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:54:58,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265756 s, 154 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:55:03,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:55:03,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789787 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:09,009::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 11:55:09,010::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:55:09,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:09,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.040095 s, 51.1 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:09,138::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.040095 s, 51.1 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 11:55:09,140::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 11:55:09,141::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 11:55:09,146::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 11:55:09,152::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:09,158::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:09,161::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:09,164::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:09,166::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:09,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-14142::WARNING::2013-03-17 11:55:09,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:55:09,545::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:55:09,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:09,669::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:09,669::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 11:55:09,672::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 11:55:09,673::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 11:55:09,673::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 11:55:09,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:10,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:10,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:10,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947995 s, 432 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:55:13,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:55:13,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151886 s, 270 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:55:19,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:55:19,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:55:20,000::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:55:20,008::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:55:20,010::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:55:20,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:20,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183884 s, 223 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:55:23,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:55:23,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145632 s, 281 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:30,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:30,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980726 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:55:33,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:55:33,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874539 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:40,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:40,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149522 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:55:43,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:55:44,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013748 s, 298 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:55:50,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:55:50,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083233 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:55:54,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:55:54,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139891 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:56:00,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:56:00,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00460674 s, 889 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:56:04,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:56:04,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139956 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:56:10,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:56:10,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842833 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:56:14,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:56:14,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000692625 s, 5.9 MB/s\n'; <rc> = 0 >Thread-14143::WARNING::2013-03-17 11:56:20,043::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:56:20,046::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:56:20,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:56:20,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956444 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:56:24,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:56:24,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825601 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:56:30,049::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:56:30,050::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:56:30,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:56:30,517::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:56:30,518::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:56:30,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:56:30,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162568 s, 252 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:56:34,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:56:34,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139911 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:56:40,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:56:40,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758719 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:56:44,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:56:44,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208429 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:56:50,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:56:50,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150405 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:56:54,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:56:54,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230846 s, 177 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:57:00,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:57:00,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100119 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:57:04,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:57:04,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076721 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:57:10,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:57:11,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846167 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:57:14,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:57:14,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222978 s, 184 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:57:21,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:57:21,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770011 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:57:24,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:57:24,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250841 s, 163 kB/s\n'; <rc> = 0 >Thread-14144::WARNING::2013-03-17 11:57:30,544::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:57:30,549::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:57:31,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:57:31,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875607 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:57:34,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:57:34,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760868 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:57:40,551::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:57:40,552::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:57:40,992::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:57:40,998::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:57:41,000::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:57:41,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:57:41,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010191 s, 402 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:57:44,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:57:44,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013084 s, 313 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:57:51,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:57:51,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844803 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:57:54,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:57:54,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139784 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:58:01,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:58:01,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148903 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:58:04,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:58:04,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295409 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:58:11,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:58:11,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153872 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:58:14,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:58:14,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737904 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:58:21,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:58:21,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151741 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:58:24,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:58:24,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144782 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:58:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:58:31,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014409 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:58:34,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:58:34,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780135 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14145::WARNING::2013-03-17 11:58:41,041::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:58:41,044::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:58:41,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:58:41,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0270001 s, 152 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:58:44,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:58:44,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149616 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 11:58:51,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 11:58:51,060::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 11:58:51,478::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 11:58:51,485::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 11:58:51,486::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 11:58:51,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:58:51,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108808 s, 376 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:58:54,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:58:54,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222955 s, 184 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:59:01,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:59:01,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892553 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:04,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:05,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280712 s, 146 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:59:11,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:59:11,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084848 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:15,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:15,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00303234 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:59:21,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:59:22,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781106 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:25,092::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 11:59:25,094::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:59:25,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:25,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128893 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:25,137::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128893 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 11:59:25,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 11:59:25,138::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 11:59:25,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 11:59:25,141::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:59:25,143::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:59:25,143::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:59:25,144::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:59:25,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 11:59:25,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:25,573::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 11:59:25,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 11:59:25,576::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 11:59:25,577::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 11:59:25,578::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 11:59:25,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:25,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:25,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:26,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180365 s, 227 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:59:32,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:59:32,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268425 s, 153 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:36,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:36,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722984 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 11:59:42,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:59:42,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921746 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:46,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:46,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133256 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14146::WARNING::2013-03-17 11:59:51,513::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 11:59:51,516::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 11:59:52,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 11:59:52,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938567 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 11:59:56,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 11:59:56,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141095 s, 2.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:00:01,517::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:00:01,519::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:00:01,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:00:02,183::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:00:02,189::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:00:02,190::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:02,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114807 s, 357 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:00:06,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:00:06,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143637 s, 285 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:12,405::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:00:12,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:00:12,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:12,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135709 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:12,440::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135709 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:00:12,440::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:00:12,440::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:00:12,442::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:00:12,443::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:12,444::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:12,444::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:12,444::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:12,445::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:12,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:12,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:12,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:00:12,837::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:00:12,840::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:00:12,841::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:00:12,841::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:00:12,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:13,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:13,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:13,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156567 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:00:16,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:00:16,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154006 s, 266 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:23,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:23,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138791 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:00:26,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:00:26,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785944 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:33,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:33,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145796 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:00:36,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:00:36,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151298 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:43,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:43,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156732 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:00:46,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:00:46,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151669 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:00:53,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:00:53,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183018 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:00:56,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:00:56,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219242 s, 187 kB/s\n'; <rc> = 0 >Thread-14147::WARNING::2013-03-17 12:01:02,228::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:01:02,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:01:03,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:01:03,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752248 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:01:06,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:01:06,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814682 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:01:12,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:01:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:01:12,690::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:01:12,696::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:01:12,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:01:13,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:01:13,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229653 s, 178 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:01:16,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:01:16,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126597 s, 324 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:01:23,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:01:23,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850365 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:01:26,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:01:26,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101682 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:01:33,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:01:33,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137486 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:01:36,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:01:36,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110029 s, 372 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:01:43,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:01:44,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157505 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:01:46,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:01:46,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163915 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:01:54,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:01:54,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143905 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:01:56,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:01:56,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822317 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:02:04,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:02:04,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136329 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:02:06,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:02:06,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791284 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14148::WARNING::2013-03-17 12:02:12,723::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:02:12,727::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:02:14,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:02:14,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817749 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:02:16,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:02:16,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817219 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:02:22,729::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:02:22,730::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:02:23,140::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:02:23,147::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:02:23,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:02:24,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:02:24,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018403 s, 223 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:02:26,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:02:26,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159787 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:02:34,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:02:34,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841271 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:02:36,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:02:36,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743813 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:02:44,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:02:44,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777727 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:02:46,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:02:46,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947384 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:02:54,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:02:54,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076452 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:02:56,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:02:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152441 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:03:04,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:03:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760115 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:03:07,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:03:07,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163761 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:03:14,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:03:14,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147983 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:03:17,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:03:17,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167423 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14149::WARNING::2013-03-17 12:03:23,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:03:23,183::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:03:24,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:03:24,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707417 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:03:27,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:03:27,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922369 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:03:33,186::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:03:33,188::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:03:33,633::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:03:33,640::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:03:33,642::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:03:34,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:03:34,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131023 s, 313 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:03:37,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:03:37,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188673 s, 217 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:03:44,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:03:44,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134079 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:03:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:03:47,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780469 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:03:54,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:03:55,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781254 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:03:57,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:03:57,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135491 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:04:05,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:04:05,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135061 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:07,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:07,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209232 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:04:15,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:04:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147682 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:17,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:17,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02481 s, 165 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:04:25,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:04:25,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017781 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:27,425::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:04:27,430::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:04:27,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:27,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762121 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:27,465::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762121 s, 2.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:04:27,466::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:04:27,466::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:04:27,468::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:04:27,470::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:04:27,470::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:04:27,471::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:04:27,471::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:04:27,472::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:04:27,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:27,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:27,909::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:04:27,910::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:04:27,912::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:04:27,914::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:04:27,914::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:04:27,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:28,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:28,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:28,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156612 s, 262 kB/s\n'; <rc> = 0 >Thread-14150::WARNING::2013-03-17 12:04:33,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:04:33,677::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:04:35,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:04:35,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848522 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:38,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:38,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085361 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:04:43,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:04:43,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:04:44,143::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:04:44,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:04:44,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:04:45,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:04:45,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0282723 s, 145 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:48,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:48,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171496 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:04:55,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:04:55,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0278022 s, 147 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:04:58,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:04:58,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101264 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:05,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:05,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0387659 s, 106 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:05:08,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:05:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001861 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:15,705::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:05:15,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:05:15,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:15,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118168 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:15,747::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118168 s, 1.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:05:15,748::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:05:15,748::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:05:15,750::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:05:15,751::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:15,751::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:15,752::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:15,752::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:15,752::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:15,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:16,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:16,159::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:16,160::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:05:16,162::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:05:16,164::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:05:16,164::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:05:16,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:16,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:16,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:16,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105532 s, 388 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:05:18,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:05:18,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178481 s, 229 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:26,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:26,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152235 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:05:28,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:05:28,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776922 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:05:36,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:36,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154401 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:05:38,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:05:38,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205488 s, 2.0 MB/s\n'; <rc> = 0 >Thread-14151::WARNING::2013-03-17 12:05:44,183::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:05:44,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:05:46,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:46,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083278 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:05:48,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:05:48,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846151 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:05:54,192::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:05:54,193::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:05:54,194::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:05:54,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:05:54,643::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:05:54,644::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:05:56,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:05:56,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231107 s, 177 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:05:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:05:58,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223161 s, 184 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:06:06,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:06:06,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861324 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:06:08,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:06:08,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015477 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:06:17,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:06:17,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119546 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:06:18,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:06:18,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890077 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:06:27,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:06:27,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722921 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:06:28,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:06:28,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860023 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:06:37,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:06:37,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146297 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:06:38,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:06:38,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157189 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:06:47,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:06:47,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166387 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:06:48,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:06:48,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202265 s, 203 kB/s\n'; <rc> = 0 >Thread-14152::WARNING::2013-03-17 12:06:54,669::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:06:54,672::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:06:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:06:57,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831179 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:06:59,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:06:59,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219729 s, 186 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:07:04,676::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:07:04,678::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:07:05,171::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:07:05,178::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:07:05,179::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:07:07,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:07:07,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025265 s, 162 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:07:09,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:07:09,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140168 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:07:17,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:07:17,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807711 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:07:19,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:07:19,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232329 s, 176 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:07:27,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:07:27,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155508 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:07:29,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:07:29,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835476 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:07:37,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:07:37,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154687 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:07:39,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:07:39,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745272 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:07:47,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:07:47,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917832 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:07:49,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:07:49,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147907 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:07:57,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:07:57,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146862 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:07:59,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:07:59,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215235 s, 190 kB/s\n'; <rc> = 0 >Thread-14153::WARNING::2013-03-17 12:08:05,206::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:08:05,211::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:08:07,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:08:07,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148234 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:08:09,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:08:09,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779237 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:08:15,216::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:08:15,217::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:08:15,643::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:08:15,649::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:08:15,651::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:08:17,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:08:17,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190304 s, 215 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:08:19,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:08:19,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.005575 s, 735 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:08:28,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:08:28,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796827 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:08:29,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:08:29,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140635 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:08:38,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:08:38,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764704 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:08:39,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:08:39,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738912 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:08:48,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:08:48,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894329 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:08:49,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:08:49,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819401 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:08:58,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:08:58,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156932 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:08:59,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:08:59,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814421 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:09:08,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:09:08,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268825 s, 152 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:09,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:09,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158152 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14154::WARNING::2013-03-17 12:09:15,677::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:09:15,681::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:09:18,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:09:18,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800201 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:19,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:19,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980711 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:09:25,683::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:09:25,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:09:26,175::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:09:26,181::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:09:26,183::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:09:28,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:09:28,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230487 s, 178 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:29,668::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:09:29,670::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:09:29,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:29,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00801803 s, 255 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:29,707::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00801803 s, 255 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:09:29,707::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:09:29,708::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:09:29,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:09:29,711::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:09:29,712::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:09:29,712::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:09:29,713::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:09:29,713::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:09:29,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:30,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:30,172::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:09:30,173::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:09:30,182::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:09:30,185::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:09:30,186::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:09:30,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:30,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:30,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:30,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00441877 s, 927 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:09:38,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:09:38,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961189 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:40,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:40,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229461 s, 179 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:09:48,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:09:48,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149758 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:09:50,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:09:50,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995468 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:09:58,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:09:58,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152091 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:10:00,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:10:00,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098427 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:10:08,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:08,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767306 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:10:10,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:10:10,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023654 s, 173 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:10:18,848::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:10:18,850::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:10:18,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:18,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000826094 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:10:18,888::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000826094 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:10:18,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:10:18,888::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:10:18,890::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:10:18,891::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:18,892::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:18,892::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:18,893::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:18,893::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:18,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:19,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:10:19,346::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:19,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:10:19,349::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:10:19,350::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:10:19,350::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:10:19,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:19,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:10:19,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:19,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00940939 s, 435 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:10:20,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:10:20,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00900752 s, 455 kB/s\n'; <rc> = 0 >Thread-14155::WARNING::2013-03-17 12:10:26,209::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:10:26,212::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:10:29,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:29,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025432 s, 161 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:10:30,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:10:30,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769283 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:10:36,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:10:36,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:10:36,695::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:10:36,702::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:10:36,703::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:10:40,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:40,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206493 s, 198 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:10:40,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:10:40,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142912 s, 287 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:10:50,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:10:50,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146449 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:10:50,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:10:51,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892608 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:11:00,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:11:00,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0264828 s, 155 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:11:01,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:11:01,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142932 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:11:10,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:11:10,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826657 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:11:11,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:11:11,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170169 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:11:20,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:11:20,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812294 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:11:21,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:11:21,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148342 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:11:30,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:11:30,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162388 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:11:31,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:11:31,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929563 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14156::WARNING::2013-03-17 12:11:36,729::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:11:36,733::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:11:40,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:11:40,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926796 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:11:41,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:11:41,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146771 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:11:46,734::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:11:46,736::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:11:46,737::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:11:47,361::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:11:47,369::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:11:47,370::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:11:50,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:11:50,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135529 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:11:51,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:11:51,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185818 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:12:00,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:12:00,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173722 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:12:01,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:12:01,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215286 s, 190 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:12:10,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:12:10,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898416 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:12:11,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:12:11,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788853 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:12:20,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:12:20,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00430486 s, 951 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:12:21,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:12:21,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800292 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:12:30,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:12:30,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152165 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:12:31,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:12:31,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025278 s, 162 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:12:40,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:12:41,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0269878 s, 152 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:12:41,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:12:41,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202502 s, 2.0 MB/s\n'; <rc> = 0 >Thread-14157::WARNING::2013-03-17 12:12:47,398::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:12:47,402::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:12:51,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:12:51,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847098 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:12:51,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:12:51,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974116 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:12:57,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:12:57,405::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:12:57,876::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:12:57,882::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:12:57,884::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:13:01,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:13:01,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263931 s, 155 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:13:01,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:13:01,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110236 s, 372 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:13:11,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:13:11,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147491 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:13:11,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:13:11,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110143 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:13:21,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:13:21,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147787 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:13:21,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:13:21,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106474 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:13:31,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:13:31,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083143 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:13:31,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:13:31,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142359 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:13:41,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:13:41,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0277394 s, 148 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:13:41,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:13:41,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016233 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:13:51,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:13:51,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791407 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:13:51,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:13:51,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853277 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14158::WARNING::2013-03-17 12:13:57,914::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:13:57,918::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:14:01,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:14:01,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164376 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:01,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:02,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786833 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:14:07,920::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:14:07,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:14:08,335::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:14:08,341::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:14:08,343::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:14:11,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:14:11,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907349 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:12,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:12,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162823 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:14:21,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:14:21,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849688 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:22,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105016 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:14:31,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:14:31,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0321872 s, 127 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:32,137::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:14:32,141::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:14:32,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:32,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000856464 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:32,172::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000856464 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:14:32,172::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:14:32,173::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:14:32,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:14:32,182::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:14:32,182::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:14:32,183::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:14:32,183::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:14:32,184::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:14:32,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:32,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:32,618::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:14:32,619::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:14:32,623::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:14:32,624::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:14:32,624::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:14:32,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:32,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:32,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:33,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138977 s, 295 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:14:42,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:14:42,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0256794 s, 160 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:43,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:43,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015039 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:14:52,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:14:52,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001738 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:14:53,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:14:53,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837936 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:02,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:02,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740831 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:15:03,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:03,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133646 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14159::WARNING::2013-03-17 12:15:08,370::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:15:08,373::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:15:12,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:12,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938698 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:15:13,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:13,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818725 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:15:18,375::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:15:18,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:15:18,815::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:15:18,821::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:15:18,823::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,393::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:15:22,399::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:15:22,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:22,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000779724 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:22,428::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000779724 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:15:22,428::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:15:22,428::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:15:22,430::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:15:22,431::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,432::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,432::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,433::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:22,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:22,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:15:22,883::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:15:22,886::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:15:22,887::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:15:22,887::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:15:22,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:23,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:23,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:23,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202254 s, 203 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:23,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144563 s, 283 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:15:33,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:33,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016659 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:33,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:33,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883801 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:15:43,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:43,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141806 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:43,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:43,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137487 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:15:53,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:15:53,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899355 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:15:53,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:15:53,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770169 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:16:03,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:16:03,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0047356 s, 865 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:16:03,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:16:03,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141911 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:16:13,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:16:13,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014555 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:16:13,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:16:13,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178904 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14160::WARNING::2013-03-17 12:16:18,856::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:16:18,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:16:23,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:16:23,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145563 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:16:23,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:16:23,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701239 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:16:28,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:16:28,867::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:16:29,293::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:16:29,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:16:29,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:16:33,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:16:33,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161051 s, 254 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:16:33,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:16:33,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187275 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:16:43,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:16:43,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911637 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:16:43,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:16:43,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887544 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:16:53,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:16:53,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158574 s, 258 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:16:53,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:16:54,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856603 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:17:03,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:17:03,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142142 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:17:04,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:17:04,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140687 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:17:13,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:17:13,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224874 s, 182 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:17:14,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:17:14,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231734 s, 177 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:17:23,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:17:23,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002142 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:17:24,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:17:24,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815527 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14161::WARNING::2013-03-17 12:17:29,333::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:17:29,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:17:33,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:17:33,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238179 s, 172 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:17:34,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:17:34,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700832 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:17:39,342::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:17:39,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:17:39,344::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:17:39,736::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:17:39,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:17:39,744::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:17:43,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:17:44,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276044 s, 148 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:17:44,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:17:44,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184253 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:17:54,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:17:54,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876091 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:17:54,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:17:54,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015017 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:18:04,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:18:04,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134236 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:18:04,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:18:04,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0342909 s, 119 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:18:14,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:18:14,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0277072 s, 148 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:18:14,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:18:14,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0279268 s, 147 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:18:24,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:18:24,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895418 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:18:24,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:18:24,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132282 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:18:34,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:18:34,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219859 s, 186 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:18:34,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:18:34,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145046 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14162::WARNING::2013-03-17 12:18:39,770::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:18:39,774::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:18:44,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:18:44,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895924 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:18:44,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:18:44,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207115 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:18:49,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:18:49,778::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:18:50,194::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:18:50,200::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:18:50,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:18:54,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:18:54,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165991 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:18:55,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:18:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994443 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:04,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:04,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777971 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:19:05,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:05,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014924 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:14,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:14,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0253811 s, 161 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:19:15,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:15,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015482 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:24,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:24,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139982 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:19:25,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:25,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149986 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:34,534::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:19:34,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:19:34,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:34,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136467 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:34,578::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136467 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:19:34,579::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:19:34,579::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:19:34,581::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:19:34,582::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:19:34,582::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:19:34,583::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:19:34,583::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:19:34,583::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:19:34,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:35,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:35,006::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:19:35,007::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:19:35,010::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:19:35,014::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:19:35,014::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:19:35,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:35,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:35,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:35,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:35,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147106 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:35,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123084 s, 333 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:19:45,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:45,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:45,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821958 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:19:45,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979328 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14163::WARNING::2013-03-17 12:19:50,232::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:19:50,236::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:19:55,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:19:55,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817717 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:19:55,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:19:55,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018877 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:20:00,238::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:20:00,239::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:20:00,685::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:20:00,692::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:20:00,693::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:20:05,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:05,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:20:05,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743211 s, 551 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:05,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012068 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:20:15,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:15,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:20:15,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143768 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:15,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156107 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:20:25,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:20:25,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784267 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:25,919::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:20:25,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:20:25,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:25,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130663 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:25,959::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130663 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:20:25,960::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:20:25,960::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:20:25,962::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:20:25,963::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:20:25,964::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:20:25,964::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:20:25,964::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:20:25,965::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:20:25,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:26,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:26,414::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:20:26,415::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:20:26,417::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:20:26,420::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:20:26,421::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:20:26,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:26,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:26,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:26,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100046 s, 409 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:20:35,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:20:35,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000681059 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:36,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:36,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924551 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:20:45,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:20:45,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133317 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:46,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:46,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757384 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:20:55,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:20:56,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016679 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:20:57,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:20:57,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146982 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14164::WARNING::2013-03-17 12:21:00,719::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:21:00,722::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:21:06,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:21:06,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0567796 s, 72.1 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:21:07,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:21:07,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869603 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:21:10,725::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:21:10,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:21:11,130::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:21:11,137::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:21:11,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:21:16,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:21:16,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170374 s, 240 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:21:17,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:21:17,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776074 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:21:26,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:21:26,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163142 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:21:27,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:21:27,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101648 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:21:36,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:21:36,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123436 s, 332 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:21:37,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:21:37,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084049 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:21:46,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:21:46,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864436 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:21:47,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:21:47,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154915 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:21:56,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:21:56,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0378082 s, 108 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:21:57,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:21:57,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146994 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:22:06,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:22:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100845 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:22:07,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:22:07,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020478 s, 200 kB/s\n'; <rc> = 0 >Thread-14165::WARNING::2013-03-17 12:22:11,170::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:22:11,174::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:22:16,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:22:16,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877518 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:22:17,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:22:17,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083963 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:22:21,176::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:22:21,177::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:22:21,597::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:22:21,603::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:22:21,605::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:22:26,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:22:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947309 s, 432 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:22:27,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:22:27,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140181 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:22:36,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:22:36,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838363 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:22:37,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:22:37,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859582 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:22:46,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:22:46,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845646 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:22:47,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:22:47,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899504 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:22:56,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:22:56,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146179 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:22:57,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:22:58,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159196 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:23:06,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:23:06,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014143 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:23:08,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:23:08,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010004 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:23:16,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:23:16,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135082 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:23:18,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:23:18,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015321 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14166::WARNING::2013-03-17 12:23:21,640::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:23:21,643::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:23:26,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:23:26,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806042 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:23:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:23:28,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798666 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:23:31,645::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:23:31,646::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:23:31,662::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:23:32,133::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:23:32,139::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:23:32,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:23:36,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:23:36,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160502 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:23:38,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:23:38,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859942 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:23:46,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:23:46,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010033 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:23:48,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:23:48,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143703 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:23:56,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:23:56,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138533 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:23:58,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:23:58,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830531 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:06,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:07,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779773 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:24:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:24:08,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897023 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:17,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:17,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161301 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:24:18,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:24:18,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000714437 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:27,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853351 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:24:28,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:24:28,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943948 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14167::WARNING::2013-03-17 12:24:32,176::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:24:32,180::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:24:37,129::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:24:37,130::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:24:37,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:37,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0221497 s, 92.5 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:37,181::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0221497 s, 92.5 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:24:37,182::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:24:37,182::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:24:37,184::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:24:37,186::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:37,187::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:37,187::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:37,188::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:37,188::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:37,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:37,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:37,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:37,621::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:24:37,624::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:24:37,625::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:24:37,626::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:24:37,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:37,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:37,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:38,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00856541 s, 478 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:24:39,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:24:39,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206196 s, 199 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:24:42,183::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:24:42,184::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:24:42,581::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:24:42,588::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:24:42,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:24:48,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:48,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173928 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:24:49,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:24:49,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0305846 s, 134 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:24:58,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:24:58,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855244 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:24:59,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:24:59,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02634 s, 156 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:25:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:25:08,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102576 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:09,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:09,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831495 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:25:18,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:25:18,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138575 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:19,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:19,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721371 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:25:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:25:28,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767775 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:29,532::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:25:29,534::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:25:29,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:29,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000751902 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:29,573::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000751902 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:25:29,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:25:29,574::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:25:29,576::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:25:29,577::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:25:29,578::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:25:29,578::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:25:29,578::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:25:29,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:25:29,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:30,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:30,006::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:25:30,007::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:25:30,009::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:25:30,011::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:25:30,012::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:25:30,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:30,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:30,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:30,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.008202 s, 499 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:25:38,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:25:38,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181684 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:40,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:40,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011471 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14168::WARNING::2013-03-17 12:25:42,617::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:25:42,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:25:48,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:25:48,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111749 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:25:50,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:25:50,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772951 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:25:52,622::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:25:52,623::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:25:53,088::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:25:53,095::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:25:53,096::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:25:58,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:25:58,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119038 s, 344 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:26:00,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:26:00,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903754 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:26:08,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:26:08,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218063 s, 188 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:26:10,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:26:10,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873818 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:26:18,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:26:18,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738602 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:26:20,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:26:20,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152738 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:26:28,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:26:28,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738108 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:26:30,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:26:30,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783773 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:26:38,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:26:38,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145896 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:26:40,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:26:40,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873916 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:26:48,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:26:48,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130631 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:26:50,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:26:51,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158945 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14169::WARNING::2013-03-17 12:26:53,121::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:26:53,125::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:26:58,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:26:58,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014565 s, 281 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:27:01,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:27:01,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0338109 s, 121 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:27:03,127::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:27:03,128::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:27:03,588::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:27:03,595::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:27:03,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:27:08,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:27:08,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113021 s, 362 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:27:11,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:27:11,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173761 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:27:18,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:27:18,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920099 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:27:21,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:27:21,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016073 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:27:28,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:27:28,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149746 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:27:31,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:27:31,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288226 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:27:38,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:27:38,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010975 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:27:41,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:27:41,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139931 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:27:48,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:27:48,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829521 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:27:51,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:27:51,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014374 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:27:58,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:27:58,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141782 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:28:01,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:28:01,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00298122 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14170::WARNING::2013-03-17 12:28:03,625::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:28:03,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:28:08,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:28:09,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144265 s, 284 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:28:11,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:28:11,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150775 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:28:13,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:28:13,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:28:14,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:28:14,069::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:28:14,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:28:19,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:28:19,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119396 s, 343 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:28:21,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:28:21,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865227 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:28:29,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:28:29,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206427 s, 198 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:28:31,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:28:31,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191789 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:28:39,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:28:39,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112348 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:28:41,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:28:42,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167883 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:28:49,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:28:49,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180392 s, 227 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:28:52,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:28:52,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167676 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:28:59,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:28:59,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132834 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:29:02,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:29:02,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146752 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:29:09,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:09,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203306 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:29:12,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:29:12,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166887 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14171::WARNING::2013-03-17 12:29:14,105::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:29:14,109::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:29:19,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:19,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913855 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:29:22,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:29:22,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160197 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:29:24,112::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:29:24,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:29:24,114::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:29:24,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:29:24,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:29:24,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:29,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:29,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190014 s, 216 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:29:32,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:29:32,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937165 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:29:39,481::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:29:39,482::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:29:39,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:39,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129649 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:29:39,529::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129649 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:29:39,530::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:29:39,530::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:29:39,531::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:29:39,532::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:39,533::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:39,534::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:39,534::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:39,534::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:39,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:29:39,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:29:39,983::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:29:39,986::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:29:39,987::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:29:39,988::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:29:39,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:40,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:29:40,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:40,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150881 s, 271 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:29:42,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:29:42,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233506 s, 175 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:29:50,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:29:50,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189875 s, 216 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:29:52,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:29:52,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000696914 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:30:00,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:30:00,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856197 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:02,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:02,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135975 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:30:10,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:30:10,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180993 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:12,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:12,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145078 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:30:20,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:30:20,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014773 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:22,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:22,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121818 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14172::WARNING::2013-03-17 12:30:24,660::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:30:24,664::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:30:30,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:30:30,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131896 s, 311 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:32,927::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:30:32,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:30:32,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:32,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000989682 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:32,956::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000989682 s, 2.1 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:30:32,956::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:30:32,957::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:30:32,958::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:30:32,959::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:30:32,960::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:30:32,961::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:30:32,961::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:30:32,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:30:32,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:33,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:33,395::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:30:33,396::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:30:33,398::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:30:33,400::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:30:33,400::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:30:33,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:33,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:33,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:33,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00904118 s, 453 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:30:34,666::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:30:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:30:35,087::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:30:35,093::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:30:35,094::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:30:40,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:30:40,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013381 s, 306 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:43,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:43,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884727 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:30:50,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:30:50,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140375 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:30:53,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:30:53,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811683 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:31:00,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:31:00,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768767 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:31:03,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:31:04,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138807 s, 295 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:31:10,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:31:10,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165045 s, 248 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:31:14,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:31:14,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846976 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:31:20,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:31:20,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139621 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:31:24,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:31:24,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118941 s, 344 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:31:30,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:31:30,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207593 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:31:34,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:31:34,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739213 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14173::WARNING::2013-03-17 12:31:35,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:31:35,132::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:31:40,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:31:40,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902941 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:31:44,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:31:44,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082279 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:31:45,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:31:45,145::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:31:45,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:31:45,619::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:31:45,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:31:51,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:31:51,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0240432 s, 170 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:31:54,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:31:54,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783897 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:32:01,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:32:01,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00653115 s, 627 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:32:04,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:32:04,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163537 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:32:11,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:32:11,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081012 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:32:14,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:32:14,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952847 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:32:21,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:32:21,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170043 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:32:24,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:32:24,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790023 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:32:31,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:32:31,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901031 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:32:35,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:32:35,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167042 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:32:41,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:32:41,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014415 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:32:45,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:32:45,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153189 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14174::WARNING::2013-03-17 12:32:45,646::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:32:45,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:32:51,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:32:51,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918762 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:32:55,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:32:55,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265288 s, 154 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:32:55,658::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:32:55,660::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:32:56,126::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:32:56,132::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:32:56,133::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:33:01,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:33:01,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136893 s, 299 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:33:05,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:33:05,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818609 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:33:11,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:33:11,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738909 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:33:15,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:33:15,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824429 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:33:21,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:33:21,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832755 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:33:25,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:33:25,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830326 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:33:31,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:33:31,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084669 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:33:35,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:33:35,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830324 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:33:41,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:33:41,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250024 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:33:45,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:33:45,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184691 s, 222 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:33:51,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:33:51,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739593 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:33:55,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:33:56,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217571 s, 188 kB/s\n'; <rc> = 0 >Thread-14175::WARNING::2013-03-17 12:33:56,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:33:56,178::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 12:34:01,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:01,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117334 s, 3.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:34:06,182::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:34:06,183::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:06,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:06,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126876 s, 323 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:34:06,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:34:06,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:34:06,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:11,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:11,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184907 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:34:16,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:16,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072576 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:21,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:21,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226622 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:34:26,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:26,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873871 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:32,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:32,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231902 s, 177 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:34:36,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:36,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942157 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:42,112::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:34:42,114::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:34:42,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:42,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00933488 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:42,194::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00933488 s, 219 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:34:42,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:34:42,194::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:34:42,196::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:34:42,197::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:42,197::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:42,198::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:42,198::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:42,198::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:42,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:42,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:42,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:34:42,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:34:42,640::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:34:42,641::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:34:42,641::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:34:42,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:42,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:42,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:43,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102213 s, 401 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:34:46,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:46,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019602 s, 209 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:34:53,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:34:53,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138318 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:34:56,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:34:56,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149305 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:35:03,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:35:03,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182492 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14176::WARNING::2013-03-17 12:35:06,694::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:35:06,699::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:35:06,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:06,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162316 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:35:13,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:35:13,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746724 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:35:16,703::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:35:16,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:35:16,705::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:16,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:16,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01257 s, 326 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:35:17,159::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:35:17,170::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:35:17,172::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:35:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:35:23,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131009 s, 313 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:26,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:26,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0255872 s, 160 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:35:33,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:35:33,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716575 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:37,005::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:35:37,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:35:37,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:37,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149043 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:37,049::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149043 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:35:37,049::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:35:37,049::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:35:37,052::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:35:37,053::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:35:37,053::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:35:37,054::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:35:37,054::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:35:37,055::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:35:37,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:37,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:37,449::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:35:37,450::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:35:37,452::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:35:37,454::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:35:37,454::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:35:37,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:37,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:37,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:37,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195448 s, 210 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:35:43,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:35:43,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0360351 s, 114 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:47,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:47,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877385 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:35:53,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:35:53,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781377 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:35:57,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:35:58,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231903 s, 177 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:36:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:36:03,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133617 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:36:08,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:36:08,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118711 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:36:13,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:36:13,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705709 s, 5.8 MB/s\n'; <rc> = 0 >Thread-14177::WARNING::2013-03-17 12:36:17,216::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:36:17,221::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:36:18,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:36:18,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211674 s, 194 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:36:23,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:36:23,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780608 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:36:27,225::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:36:27,227::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:36:27,667::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:36:27,674::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:36:27,675::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:36:28,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:36:28,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125688 s, 326 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:36:33,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:36:33,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162839 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:36:38,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:36:38,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199321 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:36:43,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:36:43,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162628 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:36:48,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:36:48,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107936 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:36:53,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:36:53,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741419 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:36:58,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:36:58,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018025 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:37:03,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:37:03,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239789 s, 171 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:37:08,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:37:08,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813738 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:37:13,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:37:13,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140689 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:37:18,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:37:18,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769392 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:37:23,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:37:23,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196735 s, 208 kB/s\n'; <rc> = 0 >Thread-14178::WARNING::2013-03-17 12:37:27,704::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:37:27,707::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:37:28,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:37:28,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079351 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:37:33,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:37:33,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831414 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:37:37,710::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:37:37,711::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:37:38,214::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:37:38,220::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:37:38,222::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:37:38,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:37:38,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146732 s, 279 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:37:43,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:37:43,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139167 s, 294 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:37:48,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:37:48,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149638 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:37:53,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:37:54,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181536 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:37:59,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:37:59,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.024993 s, 164 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:38:04,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:38:04,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257627 s, 159 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:38:09,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:38:09,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.027417 s, 149 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:38:14,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:38:14,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819043 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:38:19,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:38:19,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104422 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:38:24,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:38:24,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0266162 s, 154 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:38:29,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:38:29,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796722 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:38:34,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:38:34,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989528 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14179::WARNING::2013-03-17 12:38:38,249::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:38:38,255::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:38:39,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:38:39,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088451 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:38:44,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:38:44,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193086 s, 212 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:38:48,258::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:38:48,260::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:38:48,658::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:38:48,666::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:38:48,667::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:38:49,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:38:49,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00786986 s, 520 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:38:54,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:38:54,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147943 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:38:59,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:38:59,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691496 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:04,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:04,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161387 s, 254 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:39:09,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:39:09,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245885 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:14,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:14,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793466 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:39:19,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:39:19,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730994 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:24,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:24,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100096 s, 409 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:39:29,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:39:29,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771893 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:34,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841629 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:39:39,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:39:39,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793938 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:44,576::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:39:44,578::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:39:44,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:44,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0112226 s, 182 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:44,630::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0112226 s, 182 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:39:44,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:39:44,631::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:39:44,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:39:44,634::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:39:44,635::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:39:44,635::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:39:44,636::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:39:44,636::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:39:44,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:45,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:45,082::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:39:45,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:39:45,086::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:39:45,087::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:39:45,088::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:39:45,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:45,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:45,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:45,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0287042 s, 143 kB/s\n'; <rc> = 0 >Thread-14180::WARNING::2013-03-17 12:39:48,696::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:39:48,703::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:39:49,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:39:49,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245691 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:39:55,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:39:55,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744658 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:39:58,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:39:58,707::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:39:59,137::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:39:59,144::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:39:59,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:39:59,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:00,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00974425 s, 420 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:40:05,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:40:05,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167992 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:10,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:10,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153542 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:40:15,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:40:15,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022122 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:20,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:20,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278386 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:40:25,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:40:25,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018993 s, 216 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:30,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:30,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241617 s, 170 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:40:35,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:40:35,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129451 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:40,349::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:40:40,350::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:40:40,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:40,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000673432 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:40,384::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000673432 s, 3.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:40:40,385::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:40:40,385::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:40:40,387::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:40:40,388::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:40:40,389::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:40:40,389::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:40:40,389::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:40:40,390::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:40:40,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:40,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:40,820::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:40:40,822::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:40:40,824::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:40:40,825::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:40:40,826::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:40:40,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:41,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:41,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:41,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00367078 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:40:45,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:40:45,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190649 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:40:51,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:40:51,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000655486 s, 6.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:40:56,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:40:56,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789315 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14181::WARNING::2013-03-17 12:40:59,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:40:59,179::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:41:01,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:41:01,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924742 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:41:06,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:41:06,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100457 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:41:09,180::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:41:09,182::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:41:09,183::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:41:09,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:41:09,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:41:09,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:41:11,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:41:11,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018879 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:41:16,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:41:16,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123751 s, 331 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:41:21,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:41:21,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160554 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:41:26,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:41:26,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150903 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:41:31,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:41:31,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017074 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:41:36,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:41:36,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019175 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:41:41,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:41:41,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152271 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:41:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:41:46,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0353446 s, 116 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:41:51,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:41:51,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160581 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:41:56,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:41:56,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242476 s, 169 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:42:01,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:42:01,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145767 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:42:06,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:42:06,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080165 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14182::WARNING::2013-03-17 12:42:09,623::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:42:09,626::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:42:12,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:42:12,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974204 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:42:16,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:42:16,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00447072 s, 916 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:42:19,628::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:42:19,629::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:42:20,102::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:42:20,109::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:42:20,110::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:42:22,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:42:22,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00937463 s, 437 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:42:26,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:42:26,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900102 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:42:32,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:42:32,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846046 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:42:36,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:42:36,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131846 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:42:42,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:42:42,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835787 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:42:46,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:42:46,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235777 s, 174 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:42:52,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:42:52,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790827 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:42:56,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:42:56,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00555391 s, 737 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:43:02,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:43:02,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126726 s, 323 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:43:06,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:43:06,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808222 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:43:12,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:43:12,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133641 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:43:16,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:43:16,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893009 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14183::WARNING::2013-03-17 12:43:20,143::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:43:20,148::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:43:22,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:43:22,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810013 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:43:27,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:43:27,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897453 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:43:30,151::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:43:30,153::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:43:30,620::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:43:30,627::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:43:30,629::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:43:32,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:43:32,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130602 s, 314 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:43:37,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:43:37,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121363 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:43:42,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:43:42,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147414 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:43:47,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:43:47,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728867 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:43:52,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:43:52,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244375 s, 168 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:43:57,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:43:57,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949929 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:44:02,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:44:02,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777289 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:07,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:07,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192536 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:44:12,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:44:12,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954906 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:17,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:17,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209061 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:44:23,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:44:23,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014685 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:27,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123533 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14184::WARNING::2013-03-17 12:44:30,655::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:44:30,658::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:44:33,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:44:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767093 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:37,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:37,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751152 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:44:40,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:44:40,674::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:44:41,086::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:44:41,092::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:44:41,094::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:44:43,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:44:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172486 s, 237 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:47,509::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:44:47,510::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:44:47,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:47,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0468374 s, 43.7 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:47,601::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0468374 s, 43.7 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:44:47,601::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:44:47,602::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:44:47,603::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:44:47,604::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:44:47,605::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:44:47,605::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:44:47,606::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:44:47,606::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:44:47,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:48,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:48,040::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:44:48,041::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:44:48,044::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:44:48,045::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:44:48,046::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:44:48,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:48,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:48,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:48,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013435 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:44:53,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:44:53,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175375 s, 234 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:44:58,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:44:58,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858226 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:03,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:03,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025003 s, 164 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:45:08,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:45:08,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136988 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:13,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:13,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139893 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:45:18,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:45:18,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177259 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:23,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:23,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158395 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:45:28,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:45:28,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104147 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:33,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:33,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744538 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:45:38,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:45:38,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00400849 s, 1.0 MB/s\n'; <rc> = 0 >Thread-14185::WARNING::2013-03-17 12:45:41,119::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:45:41,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:45:43,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:45:43,655::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:45:43,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:43,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140642 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:43,701::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140642 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:45:43,701::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:45:43,701::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:45:43,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:45:43,704::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:43,705::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:43,705::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:43,706::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:43,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:43,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:44,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:44,177::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:44,178::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:45:44,181::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:45:44,182::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:45:44,182::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:45:44,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:44,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:45:44,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:44,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101493 s, 404 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:45:48,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:45:48,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809324 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:45:51,128::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:45:51,130::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:45:51,554::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:45:51,561::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:45:51,562::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:45:54,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:45:54,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00642153 s, 638 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:45:58,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:45:58,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977165 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:46:04,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:46:04,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815687 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:46:08,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:46:08,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138568 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:46:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:46:14,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289838 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:46:18,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:46:18,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188192 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:46:24,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:46:24,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905857 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:46:28,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:46:28,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0046907 s, 873 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:46:34,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:46:34,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151279 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:46:38,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:46:38,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722413 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:46:44,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:46:45,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252458 s, 162 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:46:48,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:46:48,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962911 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14186::WARNING::2013-03-17 12:46:51,588::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:46:51,592::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:46:55,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:46:55,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100331 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:46:58,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:46:58,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100462 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:47:01,594::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:47:01,595::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:47:01,596::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:47:02,078::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:47:02,085::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:47:02,086::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:47:05,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:47:05,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195482 s, 210 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:47:09,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:47:09,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719754 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:47:15,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:47:15,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137298 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:47:19,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:47:19,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136857 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:47:25,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:47:25,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268892 s, 152 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:47:29,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:47:29,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751539 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:47:35,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:47:35,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144695 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:47:39,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:47:39,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134951 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:47:45,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:47:45,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986408 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:47:49,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:47:49,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858726 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:47:55,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:47:55,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159231 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:47:59,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:47:59,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147827 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14187::WARNING::2013-03-17 12:48:02,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:48:02,131::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:48:05,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:48:05,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156004 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:48:09,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:48:09,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861395 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:48:12,134::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:48:12,135::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:48:12,605::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:48:12,611::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:48:12,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:48:15,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:48:15,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203851 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:48:19,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:48:19,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808589 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:48:25,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:48:25,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082469 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:48:29,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:48:29,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787628 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:48:35,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:48:35,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841019 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:48:39,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:48:39,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842408 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:48:45,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:48:45,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147556 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:48:49,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:48:49,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00393218 s, 1.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:48:56,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:48:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884513 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:48:59,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:48:59,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136466 s, 300 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:49:06,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:49:06,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146021 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:09,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:09,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149464 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14188::WARNING::2013-03-17 12:49:12,638::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:49:12,647::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:49:16,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:49:16,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944945 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:19,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:19,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793903 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:49:22,659::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:49:22,660::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:49:23,054::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:49:23,061::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:49:23,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:49:26,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:49:26,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188559 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:29,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:29,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152366 s, 269 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:49:36,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:49:36,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773878 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:39,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:39,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124686 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:49:46,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:49:46,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102695 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:49,725::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:49:49,726::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:49:49,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:49,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000816827 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:49,756::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000816827 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:49:49,757::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:49:49,757::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:49:49,758::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:49:49,760::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:49:49,760::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:49:49,761::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:49:49,761::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:49:49,761::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:49:49,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:50,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:50,192::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:49:50,193::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:49:50,195::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:49:50,196::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:49:50,197::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:49:50,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:50,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:49:50,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:49:50,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916549 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:49:56,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:49:56,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159867 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:50:00,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:50:00,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822576 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:06,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:06,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176676 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:50:10,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:50:10,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167773 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:16,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:16,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009067 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:50:20,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:50:20,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969357 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14189::WARNING::2013-03-17 12:50:23,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:50:23,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:50:26,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:26,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251801 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:50:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:50:30,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919952 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:50:33,100::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:50:33,101::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:50:33,584::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:50:33,591::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:50:33,592::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:36,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:36,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112988 s, 363 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:50:40,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:50:40,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160399 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:46,914::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:50:46,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:50:46,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:46,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144009 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:46,948::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144009 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:50:46,949::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:50:46,949::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:50:46,951::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:50:46,953::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:46,954::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:46,954::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:46,955::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:46,955::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:46,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:47,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:47,389::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:50:47,390::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:50:47,392::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:50:47,394::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:50:47,394::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:50:47,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:47,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:47,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:47,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153525 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:50:50,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:50:50,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893924 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:50:57,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:50:57,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0259464 s, 158 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:51:00,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:51:00,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802619 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:51:07,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:51:07,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723137 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:51:10,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:51:10,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132762 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:51:17,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:51:18,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138899 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:51:20,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:51:20,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141496 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:51:28,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:51:28,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160861 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:51:30,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:51:31,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114211 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14190::WARNING::2013-03-17 12:51:33,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:51:33,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:51:38,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:51:38,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833612 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:51:41,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:51:41,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831052 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:51:43,621::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:51:43,622::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:51:44,146::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:51:44,152::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:51:44,154::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:51:48,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:51:48,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176231 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:51:51,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:51:51,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101424 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:51:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:51:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799084 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:52:01,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:52:01,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946564 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:52:08,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:52:08,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147181 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:52:11,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:52:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140206 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:52:18,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:52:18,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939168 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:52:21,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:52:21,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146195 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:52:28,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:52:28,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911144 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:52:31,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:52:31,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165916 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:52:38,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:52:38,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167397 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:52:41,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:52:41,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883147 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14191::WARNING::2013-03-17 12:52:44,179::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:52:44,182::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:52:48,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:52:48,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892438 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:52:51,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:52:51,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756665 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:52:54,191::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:52:54,192::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:52:54,193::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:52:54,698::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:52:54,704::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:52:54,706::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:52:58,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:52:58,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226819 s, 181 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:53:01,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:53:01,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931025 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:53:08,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:53:08,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091858 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:53:11,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:53:11,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866101 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:53:18,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:53:19,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143622 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:53:21,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:53:21,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125632 s, 326 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:53:29,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:53:29,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884522 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:53:31,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:53:31,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167943 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:53:39,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:53:39,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148678 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:53:41,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:53:41,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00763149 s, 537 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:53:49,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:53:49,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132927 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:53:51,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:53:51,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253071 s, 1.6 MB/s\n'; <rc> = 0 >Thread-14192::WARNING::2013-03-17 12:53:54,735::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:53:54,739::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:53:59,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:53:59,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00682739 s, 600 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:01,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:01,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190909 s, 2.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:54:04,741::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:54:04,742::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:54:05,273::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:54:05,280::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:54:05,281::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:54:09,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:54:09,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227523 s, 180 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:11,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:11,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159353 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:54:19,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:54:19,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107816 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:21,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:21,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146826 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:54:29,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:54:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166945 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:31,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:31,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901458 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:54:39,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:54:39,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780702 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:41,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:41,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199073 s, 206 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:54:49,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:54:49,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163415 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:51,945::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:54:51,947::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:54:51,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:51,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000741928 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:51,977::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000741928 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:54:51,978::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:54:51,978::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:54:51,980::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:54:51,981::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:54:51,981::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:54:51,982::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:54:51,982::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:54:51,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:54:51,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:52,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:52,409::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:54:52,410::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:54:52,412::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:54:52,414::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:54:52,414::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:54:52,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:54:52,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:54:52,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142017 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:54:59,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:54:59,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144564 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:55:02,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:55:02,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140207 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14193::WARNING::2013-03-17 12:55:05,309::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:55:05,313::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:55:09,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:09,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826304 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:55:12,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:55:12,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787621 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:55:15,316::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:55:15,317::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:55:15,805::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:55:15,812::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:55:15,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:19,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:19,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025063 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:55:22,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:55:22,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758888 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:55:30,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:30,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137074 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:55:32,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:55:33,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013453 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:55:40,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:40,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904365 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:55:43,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:55:43,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855117 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:55:50,177::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 12:55:50,178::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:55:50,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:50,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000891106 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:55:50,218::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000891106 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 12:55:50,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 12:55:50,219::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 12:55:50,220::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 12:55:50,221::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:50,222::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:50,222::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:50,223::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:50,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:50,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:50,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:55:50,656::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:55:50,657::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 12:55:50,660::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 12:55:50,661::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 12:55:50,661::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 12:55:50,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:51,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:55:51,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:55:51,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638279 s, 642 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:55:53,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:55:53,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075371 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:56:01,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:56:01,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144513 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:56:03,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:56:03,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226749 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:56:11,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:56:11,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108091 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:56:13,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:56:13,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178317 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14194::WARNING::2013-03-17 12:56:15,852::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:56:15,858::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:56:21,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:56:21,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819182 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:56:23,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:56:23,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800701 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:56:25,863::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:56:25,864::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:56:26,332::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:56:26,339::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:56:26,340::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:56:31,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:56:31,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00589751 s, 695 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:56:33,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:56:33,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792761 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:56:41,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:56:41,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926668 s, 442 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:56:43,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:56:43,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191704 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:56:51,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:56:51,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884632 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:56:53,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:56:53,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792271 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:57:01,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:57:01,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0248461 s, 165 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:57:03,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:57:03,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143537 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:57:11,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:57:11,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166857 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:57:13,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:57:13,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118298 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:57:21,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:57:21,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131668 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:57:23,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:57:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012817 s, 3.2 MB/s\n'; <rc> = 0 >Thread-14195::WARNING::2013-03-17 12:57:26,367::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:57:26,370::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:57:31,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:57:31,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981634 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:57:33,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:57:33,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814931 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:57:36,373::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:57:36,374::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:57:36,776::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:57:36,782::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:57:36,783::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:57:41,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:57:41,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078357 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:57:43,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:57:43,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800225 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:57:51,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:57:52,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847737 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:57:53,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:57:53,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147843 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:58:02,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:58:02,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158674 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:58:03,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:58:03,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144592 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:58:12,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:58:12,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752143 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:58:13,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:58:13,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160886 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:58:22,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:58:22,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924621 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:58:23,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:58:23,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145586 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:58:32,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:58:32,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014934 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:58:33,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:58:33,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771628 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14196::WARNING::2013-03-17 12:58:36,808::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:58:36,812::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:58:42,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:58:42,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747693 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:58:43,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:58:43,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161777 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:58:46,814::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 12:58:46,815::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:58:46,816::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:58:47,271::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:58:47,278::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:58:47,280::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 12:58:52,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:58:52,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244824 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:58:53,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:58:53,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806373 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:59:02,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:59:02,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085993 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:03,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:04,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107243 s, 382 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:59:12,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:59:12,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280385 s, 146 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:14,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:14,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782136 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:59:22,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:59:22,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245867 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:24,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011246 s, 364 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:59:32,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:59:32,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:34,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:34,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155791 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 12:59:42,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:59:42,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154943 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:44,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:44,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145624 s, 281 kB/s\n'; <rc> = 0 >Thread-14197::WARNING::2013-03-17 12:59:47,316::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 12:59:47,321::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 12:59:52,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 12:59:53,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813741 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:54,243::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 12:59:54,249::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:59:54,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:54,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00182161 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:54,280::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00182161 s, 1.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 12:59:54,281::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 12:59:54,281::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 12:59:54,283::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 12:59:54,284::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:59:54,284::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:59:54,285::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:59:54,285::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:59:54,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 12:59:54,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:54,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:54,705::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 12:59:54,706::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 12:59:54,709::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 12:59:54,710::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 12:59:54,710::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 12:59:54,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 12:59:55,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 12:59:55,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767865 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 12:59:57,324::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 12:59:57,326::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 12:59:57,724::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 12:59:57,730::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 12:59:57,732::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:03,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:03,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867896 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:00:05,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:00:05,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165626 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:13,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:13,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015228 s, 269 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:00:15,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:00:15,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148299 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:23,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:23,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796287 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:00:25,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:00:25,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145506 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:33,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:33,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090822 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:00:35,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:00:35,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849747 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:43,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:43,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151477 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:00:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:00:45,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743595 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:53,435::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:00:53,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:00:53,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:53,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0281399 s, 72.8 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:53,509::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0281399 s, 72.8 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:00:53,509::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:00:53,510::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:00:53,511::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:00:53,512::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:53,515::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:53,515::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:53,515::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:53,516::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:53,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:53,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:53,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:00:53,931::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:00:53,934::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:00:53,935::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:00:53,936::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:00:53,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:54,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:00:54,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:00:54,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111608 s, 367 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:00:55,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:00:55,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199823 s, 205 kB/s\n'; <rc> = 0 >Thread-14198::WARNING::2013-03-17 13:00:57,761::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:00:57,767::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:01:04,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:01:04,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0261172 s, 157 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:01:05,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:01:05,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961918 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:01:07,771::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:01:07,772::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:01:08,188::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:01:08,194::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:01:08,196::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:01:14,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:01:14,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778253 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:01:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:01:15,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215492 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:01:24,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:01:24,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126084 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:01:25,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:01:25,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0036268 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:01:34,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:01:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141524 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:01:35,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:01:35,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710283 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:01:44,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:01:44,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820026 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:01:45,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:01:45,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146196 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:01:54,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:01:54,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118927 s, 344 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:01:55,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:01:55,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242049 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:02:04,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:02:04,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725605 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:02:05,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:02:05,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856044 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14199::WARNING::2013-03-17 13:02:08,220::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:02:08,224::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:02:14,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:02:15,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182972 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:02:15,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:02:15,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217916 s, 1.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:02:18,227::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:02:18,232::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:02:18,654::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:02:18,661::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:02:18,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:02:25,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:02:25,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267434 s, 153 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:02:25,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:02:25,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834756 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:02:35,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:02:35,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823899 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:02:35,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:02:35,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100775 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:02:45,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:02:45,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139107 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:02:45,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:02:45,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903304 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:02:55,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:02:55,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139249 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:02:55,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:02:55,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097013 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:03:05,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:03:05,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108039 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:03:05,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:03:05,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130684 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:03:15,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:03:15,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0278249 s, 147 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:03:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:03:15,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089254 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14200::WARNING::2013-03-17 13:03:18,700::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:03:18,704::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:03:25,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:03:25,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859538 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:03:25,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:03:25,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828669 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:03:28,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:03:28,708::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:03:29,122::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:03:29,129::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:03:29,131::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:03:35,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:03:35,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252901 s, 162 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:03:35,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:03:36,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107259 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:03:45,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:03:45,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243423 s, 168 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:03:46,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:03:46,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888183 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:03:55,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:03:55,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127771 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:03:56,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:03:56,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00668264 s, 613 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:04:05,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:04:05,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00237918 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:06,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:06,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147354 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:04:15,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:04:16,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169077 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:16,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:16,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138693 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:04:26,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:04:26,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125503 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:26,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:26,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159589 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14201::WARNING::2013-03-17 13:04:29,155::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:04:29,159::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:04:36,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:04:36,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150405 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:36,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:36,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010047 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:04:39,160::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:04:39,161::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:04:39,162::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:04:39,577::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:04:39,585::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:04:39,586::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:04:46,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:04:46,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000683797 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:46,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166642 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:04:56,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:04:56,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148028 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:56,419::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:04:56,421::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:04:56,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:56,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000872818 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:56,447::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000872818 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:04:56,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:04:56,448::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:04:56,450::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:04:56,451::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:04:56,452::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:04:56,452::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:04:56,453::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:04:56,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:04:56,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:56,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:56,871::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:04:56,873::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:04:56,875::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:04:56,876::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:04:56,877::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:04:56,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:57,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:04:57,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:04:57,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102566 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:06,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:06,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759251 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:05:07,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:07,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177793 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:16,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103978 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:05:17,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:17,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941129 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:26,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870293 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:05:27,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:27,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151823 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:36,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:36,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092632 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:05:37,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:37,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159326 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14202::WARNING::2013-03-17 13:05:39,614::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:05:39,618::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:05:46,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:46,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806905 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:05:47,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:47,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078098 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:05:49,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:05:49,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:05:50,079::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:05:50,086::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:05:50,087::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:56,780::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:05:56,783::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:05:56,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:56,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000890521 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:56,819::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000890521 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:05:56,819::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:05:56,820::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:05:56,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:05:56,823::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:56,824::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:56,824::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:56,824::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:56,825::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:56,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:57,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:57,247::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:05:57,248::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:05:57,250::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:05:57,252::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:05:57,252::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:05:57,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:57,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:05:57,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152305 s, 269 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:57,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:05:57,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:05:57,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162748 s, 252 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:06:07,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:06:07,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095965 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:06:07,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:06:07,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793073 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:06:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:06:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160358 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:06:17,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:06:17,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855781 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:06:27,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:06:27,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157793 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:06:27,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:06:27,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770858 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:06:37,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:06:37,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136184 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:06:38,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:06:38,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811124 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:06:47,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:06:47,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137606 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:06:48,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:06:48,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155217 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14203::WARNING::2013-03-17 13:06:50,116::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:06:50,120::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:06:57,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:06:57,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149149 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:06:58,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:06:58,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265147 s, 154 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:07:00,130::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:07:00,132::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:07:00,603::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:07:00,610::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:07:00,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:07:07,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:07:07,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158752 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:07:08,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:07:08,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159378 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:07:17,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:07:17,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853036 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:07:18,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:07:18,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929354 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:07:28,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:07:28,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824039 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:07:28,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:07:28,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00254785 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:07:38,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:07:38,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147396 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:07:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:07:38,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189021 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:07:48,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:07:48,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161413 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:07:48,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:07:48,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0253896 s, 161 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:07:58,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:07:58,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772076 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:07:58,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:07:58,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875055 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14204::WARNING::2013-03-17 13:08:00,636::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:08:00,640::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:08:08,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:08:08,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00283173 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:08:08,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:08:08,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719923 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:08:10,645::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:08:10,646::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:08:11,099::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:08:11,106::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:08:11,107::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:08:18,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:08:18,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073185 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:08:18,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:08:18,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142033 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:08:28,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:08:28,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158952 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:08:29,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:08:29,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789995 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:08:38,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:08:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001271 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:08:39,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:08:39,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829952 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:08:48,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:08:48,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141168 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:08:49,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:08:49,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265382 s, 154 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:08:58,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:08:58,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00328042 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:08:59,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:08:59,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0071791 s, 571 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:08,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:08,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019602 s, 209 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:09:09,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:09:09,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102418 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14205::WARNING::2013-03-17 13:09:11,130::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:09:11,134::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:09:18,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:18,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105511 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:09:19,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:09:19,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839487 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:09:21,168::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:09:21,170::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:09:21,562::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:09:21,569::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:09:21,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:28,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:28,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150237 s, 273 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:09:29,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:09:29,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880752 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:38,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:38,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143457 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:09:39,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:09:39,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160796 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:48,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:48,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768053 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:09:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:09:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015882 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:58,775::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:09:58,777::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:09:58,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:58,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00100325 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:58,809::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00100325 s, 2.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:09:58,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:09:58,809::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:09:58,811::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:09:58,812::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:58,813::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:58,813::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:58,813::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:58,814::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:58,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:59,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:59,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:09:59,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:09:59,260::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:09:59,261::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:09:59,262::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:09:59,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:59,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:09:59,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:09:59,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028079 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:09:59,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:09:59,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135572 s, 302 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:10:09,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:10:09,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142056 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:10:09,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:10:09,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085805 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:10:19,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:10:19,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011466 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:10:19,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:10:19,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188659 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14206::WARNING::2013-03-17 13:10:21,606::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:10:21,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:10:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:10:29,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914974 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:10:30,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:10:30,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849962 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:10:31,623::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:10:31,624::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:10:31,625::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:10:32,067::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:10:32,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:10:32,075::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:10:39,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:10:39,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806492 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:10:40,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:10:40,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821996 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:10:49,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:10:49,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134506 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:10:50,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:10:50,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862167 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:10:59,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:11:00,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00306135 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:00,216::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:11:00,217::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:11:00,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:00,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00101762 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:00,250::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00101762 s, 2.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:11:00,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:11:00,250::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:11:00,252::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:11:00,253::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:11:00,254::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:11:00,254::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:11:00,254::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:11:00,255::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:11:00,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:00,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:00,678::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:11:00,679::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:11:00,682::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:11:00,683::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:11:00,683::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:11:00,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:01,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:01,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:01,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158609 s, 258 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:11:10,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:11:10,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726148 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:11,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:11,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233608 s, 175 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:11:20,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:11:20,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749701 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:21,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:21,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152072 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:11:30,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:11:30,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259776 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:31,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:31,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0428388 s, 95.6 kB/s\n'; <rc> = 0 >Thread-14207::WARNING::2013-03-17 13:11:32,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:11:32,109::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:11:40,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:11:40,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151448 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:41,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:41,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119704 s, 342 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:11:42,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:11:42,114::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:11:42,557::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:11:42,564::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:11:42,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:11:50,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:11:50,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780365 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:11:51,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:11:51,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025953 s, 158 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:12:00,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:12:00,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754197 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:12:01,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:12:02,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794557 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:12:10,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:12:10,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907956 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:12:12,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:12:12,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087557 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:12:20,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:12:20,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073991 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:12:22,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:12:22,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136421 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:12:30,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:12:30,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142862 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:12:32,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:12:32,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823113 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:12:40,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:12:40,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169043 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:12:42,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:12:42,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00539808 s, 759 kB/s\n'; <rc> = 0 >Thread-14208::WARNING::2013-03-17 13:12:42,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:12:42,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:12:50,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:12:50,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774528 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:12:52,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 13:12:52,653::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:12:52,656::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 13:12:52,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280875 s, 146 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:12:53,102::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:12:53,107::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:12:53,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:13:00,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:13:00,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887712 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:13:02,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:13:02,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242902 s, 169 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:13:10,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:13:10,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270178 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:13:12,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:13:12,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250243 s, 164 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:13:20,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:13:20,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169817 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:13:22,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:13:22,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942602 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:13:30,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:13:30,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821354 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:13:33,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:13:33,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749185 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:13:40,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:13:40,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161779 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:13:43,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:13:43,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755394 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:13:50,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:13:50,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104106 s, 3.9 MB/s\n'; <rc> = 0 >Thread-14209::WARNING::2013-03-17 13:13:53,135::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:13:53,139::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:13:53,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:13:53,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101153 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:14:00,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:14:00,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015028 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:14:03,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:14:03,143::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:03,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:03,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00818594 s, 500 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:14:03,593::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:14:03,598::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:14:03,600::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:14:10,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:14:10,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244262 s, 168 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:14:13,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:13,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163579 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:14:20,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:14:20,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140845 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:14:23,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:23,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379034 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:14:30,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:14:30,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821358 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:14:33,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:33,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160269 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:14:40,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:14:40,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134088 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:14:43,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:43,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762714 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:14:51,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:14:51,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164311 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:14:53,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:14:53,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133909 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:01,049::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:15:01,051::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:15:01,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:01,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0189236 s, 108 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:01,114::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0189236 s, 108 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:15:01,114::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:15:01,115::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:15:01,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:15:01,126::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:01,129::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:01,130::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:01,131::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:01,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:01,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:01,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:01,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:01,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:15:01,578::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:15:01,579::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:15:01,580::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:15:01,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:01,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:01,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:01,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165755 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14210::WARNING::2013-03-17 13:15:03,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:15:03,629::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:15:03,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:03,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128632 s, 318 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:12,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:12,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018098 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:15:13,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:15:13,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:13,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:13,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116831 s, 351 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:15:14,095::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:15:14,101::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:15:14,102::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:15:22,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:22,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862168 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:15:23,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:23,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151782 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:32,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:32,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813585 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:15:34,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:34,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823539 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:42,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:42,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785654 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:15:44,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:44,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00999256 s, 410 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:15:52,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:15:52,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771143 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:15:54,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:15:54,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879649 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:16:02,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:16:02,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147576 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:04,234::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:16:04,236::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:16:04,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:04,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00563145 s, 364 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:04,279::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00563145 s, 364 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:16:04,279::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:16:04,280::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:16:04,281::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:16:04,282::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:04,284::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:04,284::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:04,285::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:04,285::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:04,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:04,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:04,719::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:04,720::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:16:04,722::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:16:04,723::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:16:04,724::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:16:04,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:05,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:05,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00690844 s, 593 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:16:12,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:16:12,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753754 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14211::WARNING::2013-03-17 13:16:14,128::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:16:14,135::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:16:15,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:15,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127536 s, 321 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:16:22,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:16:22,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731004 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:16:24,141::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:16:24,142::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:16:24,144::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:16:24,566::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:16:24,572::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:16:24,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:16:25,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:25,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133334 s, 307 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:16:32,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:16:32,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100962 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:35,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:35,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000694773 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:16:42,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:16:42,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816536 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:45,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:45,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087992 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:16:52,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:16:52,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939699 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:16:55,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:16:55,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853211 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:17:02,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:17:02,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143128 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:17:05,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:17:05,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691728 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:17:12,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:17:12,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861664 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:17:15,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:17:15,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133811 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:17:22,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:17:22,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133298 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14212::WARNING::2013-03-17 13:17:24,610::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:17:24,617::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:17:25,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:17:25,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267315 s, 153 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:17:32,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:17:32,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893487 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:17:34,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:17:34,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:17:35,074::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:17:35,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:17:35,082::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:17:35,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:17:35,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198045 s, 207 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:17:42,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:17:42,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135113 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:17:45,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:17:45,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806957 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:17:52,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:17:52,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140717 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:17:56,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:17:56,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792877 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:18:02,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:18:02,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816146 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:18:06,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:18:06,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780183 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:18:12,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:18:12,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899812 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:18:16,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:18:16,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913985 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:18:22,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:18:22,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746502 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:18:26,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:18:26,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124837 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:18:32,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:18:32,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131451 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14213::WARNING::2013-03-17 13:18:35,107::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:18:35,114::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:18:36,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:18:36,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821399 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:18:42,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:18:42,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853743 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:18:45,117::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:18:45,118::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:18:45,590::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:18:45,597::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:18:45,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:18:46,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:18:46,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208663 s, 196 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:18:52,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:18:52,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124753 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:18:56,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:18:56,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0270349 s, 152 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:19:02,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:19:03,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145286 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:19:06,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:19:06,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178533 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:19:13,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:19:13,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901038 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:19:16,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:19:16,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156141 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:19:23,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:19:23,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774232 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:19:26,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:19:26,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759368 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:19:33,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:19:33,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809793 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:19:36,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:19:36,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142616 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:19:43,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:19:43,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102429 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14214::WARNING::2013-03-17 13:19:45,623::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:19:45,627::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:19:46,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:19:46,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937243 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:19:53,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:19:53,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957376 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:19:55,629::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:19:55,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:19:56,052::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:19:56,058::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:19:56,060::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:19:56,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:19:57,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216329 s, 189 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:03,222::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:20:03,224::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:20:03,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:03,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0177896 s, 115 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:03,271::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0177896 s, 115 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:20:03,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:20:03,272::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:20:03,276::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:20:03,277::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:20:03,278::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:20:03,278::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:20:03,279::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:20:03,279::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:20:03,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:03,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:03,702::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:20:03,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:20:03,705::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:20:03,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:20:03,707::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:20:03,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:04,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:04,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:04,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177236 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:20:07,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:20:07,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01842 s, 222 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:14,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:14,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876826 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:20:17,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:20:17,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839382 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:24,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:24,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010223 s, 401 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:20:27,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:20:27,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139256 s, 294 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:34,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:34,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082582 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:20:37,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:20:37,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140995 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:44,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:44,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107038 s, 383 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:20:47,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:20:47,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085538 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:20:54,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:20:54,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00391821 s, 1.0 MB/s\n'; <rc> = 0 >Thread-14215::WARNING::2013-03-17 13:20:56,086::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:20:56,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:20:57,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:20:57,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161407 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:21:04,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:21:04,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891579 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:21:06,092::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:21:06,093::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:21:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:21:06,505::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:21:06,506::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:07,599::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:21:07,602::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:21:07,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:07,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0088508 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:07,641::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0088508 s, 231 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:21:07,642::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:21:07,642::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:21:07,644::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:21:07,645::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:07,646::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:07,646::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:07,647::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:07,647::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:07,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:08,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:08,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:21:08,037::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:21:08,039::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:21:08,041::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:21:08,041::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:21:08,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:08,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:08,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:08,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121509 s, 337 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:21:14,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:21:14,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809044 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:18,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:18,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143044 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:21:24,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:21:24,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765646 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:28,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:28,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0258624 s, 158 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:21:34,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:21:34,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014311 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:38,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:38,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163872 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:21:44,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:21:44,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919421 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:48,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:48,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200173 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:21:54,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:21:54,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778423 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:21:58,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:21:58,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145935 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:22:04,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:22:04,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120224 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14216::WARNING::2013-03-17 13:22:06,538::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:22:06,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:22:08,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:22:08,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185667 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:22:14,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:22:14,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075536 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:22:16,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:22:16,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:22:16,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:22:16,994::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:22:17,000::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:22:17,002::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:22:19,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:22:19,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112481 s, 364 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:22:24,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:22:24,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918113 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:22:29,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:22:29,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753754 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:22:34,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:22:34,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162532 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:22:39,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:22:39,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204866 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:22:44,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:22:44,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142888 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:22:49,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:22:49,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123846 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:22:54,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:22:54,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749202 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:22:59,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:22:59,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013204 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:23:04,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:23:04,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166182 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:23:09,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:23:09,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752746 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:23:14,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:23:14,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107696 s, 3.8 MB/s\n'; <rc> = 0 >Thread-14217::WARNING::2013-03-17 13:23:17,030::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:23:17,036::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:23:19,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:23:19,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155331 s, 264 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:23:24,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:23:25,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180874 s, 226 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:23:27,038::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:23:27,040::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:23:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:23:27,484::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:23:27,485::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:23:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:23:29,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185003 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:23:35,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:23:35,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768806 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:23:39,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:23:39,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854727 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:23:45,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:23:45,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912218 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:23:49,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:23:49,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876446 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:23:55,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:23:55,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910113 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:23:59,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:23:59,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986758 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:24:05,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:24:05,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251015 s, 163 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:24:09,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:24:09,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098623 s, 415 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:24:15,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:24:15,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135632 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:24:19,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:24:20,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134806 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:24:25,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:24:25,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980768 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14218::WARNING::2013-03-17 13:24:27,513::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:24:27,515::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:24:30,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:24:30,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836318 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:24:35,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:24:35,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192169 s, 213 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:24:37,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:24:37,519::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:24:37,958::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:24:37,965::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:24:37,967::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:24:40,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:24:40,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0248063 s, 165 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:24:45,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:24:45,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169811 s, 241 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:24:50,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:24:50,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125068 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:24:55,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:24:55,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171164 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:25:00,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:25:00,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163332 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:05,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:25:05,596::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:25:05,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:05,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000821631 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:05,625::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000821631 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:25:05,625::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:25:05,625::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:25:05,628::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:25:05,629::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:25:05,630::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:25:05,630::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:25:05,631::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:25:05,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:25:05,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:06,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:06,053::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:25:06,054::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:25:06,056::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:25:06,057::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:25:06,058::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:25:06,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:06,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:06,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:06,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097739 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:25:10,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:25:10,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191343 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:16,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:16,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859288 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:25:20,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:25:20,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139841 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:26,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:26,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094303 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:25:30,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:25:30,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134864 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:36,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:36,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742109 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14219::WARNING::2013-03-17 13:25:38,000::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:25:38,004::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:25:40,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:25:40,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912317 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:46,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:46,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809108 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:25:48,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:25:48,008::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:25:48,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:25:48,508::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:25:48,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:25:50,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:25:50,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202926 s, 202 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:25:56,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:25:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862039 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:00,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:00,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147229 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:26:06,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:26:06,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016029 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:10,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:26:10,901::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:26:10,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:10,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000782085 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:10,934::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000782085 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:26:10,935::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:26:10,935::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:26:10,937::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:26:10,938::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:26:10,939::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:26:10,939::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:26:10,939::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:26:10,940::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:26:10,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:11,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:11,374::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:26:11,376::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:26:11,381::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:26:11,382::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:26:11,383::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:26:11,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:11,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:11,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:11,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00544802 s, 752 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:26:16,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:26:16,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943416 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:21,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:21,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863218 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:26:26,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:26:26,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149453 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:31,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:31,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141971 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:26:36,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:26:37,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140648 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:26:41,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:42,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011222 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:26:47,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:26:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777581 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14220::WARNING::2013-03-17 13:26:48,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:26:48,542::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:26:52,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:26:52,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116087 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:26:57,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:26:57,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168011 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:26:58,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:26:58,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:26:59,054::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:26:59,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:26:59,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:27:02,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:27:02,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243707 s, 168 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:27:07,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:27:07,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884258 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:27:12,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:27:12,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081303 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:27:17,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:27:17,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825241 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:27:22,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:27:22,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206494 s, 198 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:27:27,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:27:27,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075924 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:27:32,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:27:32,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145323 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:27:37,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:27:37,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151196 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:27:42,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:27:42,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152021 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:27:47,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:27:47,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155545 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:27:52,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:27:52,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191293 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:27:57,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:27:57,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103152 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14221::WARNING::2013-03-17 13:27:59,093::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:27:59,101::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:28:02,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:28:02,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757455 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:28:07,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:28:07,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884314 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:28:09,113::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:28:09,114::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:28:09,115::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:28:09,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:28:09,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:28:09,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:28:12,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:28:12,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0452542 s, 90.5 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:28:17,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:28:17,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798327 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:28:22,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:28:22,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817948 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:28:27,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:28:27,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843744 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:28:32,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:28:32,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851431 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:28:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:28:37,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159187 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:28:42,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:28:43,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784305 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:28:47,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:28:47,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157019 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:28:53,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:28:53,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891364 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:28:57,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:28:57,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845102 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:29:03,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:29:03,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128673 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:29:07,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:29:07,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012519 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14222::WARNING::2013-03-17 13:29:09,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:29:09,595::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:29:13,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:29:13,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0329369 s, 124 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:29:17,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:29:17,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884505 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:29:19,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:29:19,605::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:29:20,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:29:20,069::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:29:20,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:29:23,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:29:23,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207759 s, 197 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:29:27,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:29:27,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183573 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:29:33,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:29:33,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00671105 s, 610 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:29:37,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:29:37,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145392 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:29:43,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:29:43,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220552 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:29:47,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:29:47,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834865 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:29:53,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:29:53,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150573 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:29:57,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:29:57,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163705 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:30:03,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:30:03,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00529285 s, 774 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:07,977::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:30:07,978::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:30:07,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:08,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129486 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:08,016::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129486 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:30:08,016::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:30:08,017::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:30:08,018::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:30:08,019::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:30:08,020::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:30:08,020::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:30:08,021::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:30:08,021::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:30:08,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:08,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:08,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:30:08,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:30:08,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:30:08,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:30:08,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:30:08,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:08,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:08,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:08,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108169 s, 379 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:30:13,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:30:13,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289682 s, 141 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:18,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012931 s, 317 kB/s\n'; <rc> = 0 >Thread-14223::WARNING::2013-03-17 13:30:20,102::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:30:20,106::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:30:23,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:30:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197774 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:29,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:29,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821989 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:30:30,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:30:30,110::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:30:30,511::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:30:30,518::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:30:30,519::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:30:33,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:30:33,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185426 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:39,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:39,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134446 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:30:44,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:30:44,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115183 s, 356 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:49,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774513 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:30:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:30:54,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00909851 s, 450 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:30:59,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:30:59,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743657 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:04,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:04,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244859 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:31:09,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:31:09,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882338 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:14,292::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:31:14,294::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:31:14,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152518 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:14,338::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152518 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:31:14,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:31:14,338::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:31:14,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:31:14,341::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:14,342::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:14,342::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:14,343::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:14,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:14,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:14,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:14,766::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:14,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:31:14,770::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:31:14,771::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:31:14,771::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:31:14,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:15,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:15,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00960234 s, 427 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:31:19,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:31:19,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930678 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:25,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:25,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131558 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:31:29,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:31:29,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00543339 s, 754 kB/s\n'; <rc> = 0 >Thread-14224::WARNING::2013-03-17 13:31:30,567::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:31:30,576::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:31:35,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:35,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860865 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:31:39,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:31:39,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00641231 s, 639 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:31:40,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:31:40,580::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:31:40,982::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:31:40,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:31:40,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:31:45,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:45,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227241 s, 180 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:31:49,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:31:49,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128808 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:31:55,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:31:55,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135473 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:31:59,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:31:59,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079512 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:32:05,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:32:05,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130046 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:32:09,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:32:09,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00465758 s, 879 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:32:15,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:32:15,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726189 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:32:19,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:32:19,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936431 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:32:25,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:32:25,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139263 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:32:29,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:32:29,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00219313 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:32:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:32:35,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146999 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:32:39,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:32:39,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0044092 s, 929 kB/s\n'; <rc> = 0 >Thread-14225::WARNING::2013-03-17 13:32:41,016::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:32:41,020::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:32:45,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:32:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155917 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:32:49,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:32:49,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859973 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:32:51,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:32:51,024::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:32:51,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:32:51,468::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:32:51,469::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:32:56,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:32:56,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244476 s, 168 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:32:59,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:33:00,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778978 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:33:06,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:33:06,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748177 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:33:10,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:33:10,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838059 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:33:16,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:33:16,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250834 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:33:20,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:33:20,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858058 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:33:26,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:33:26,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108176 s, 379 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:33:30,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:33:30,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733563 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:33:36,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:33:36,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916908 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:33:40,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:33:40,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097063 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:33:46,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:33:46,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146284 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:33:50,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:33:50,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724041 s, 5.7 MB/s\n'; <rc> = 0 >Thread-14226::WARNING::2013-03-17 13:33:51,497::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:33:51,505::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:33:56,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:33:56,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254711 s, 161 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:34:00,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:34:00,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075417 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:34:01,513::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:34:01,514::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:34:01,515::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:34:02,032::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:34:02,038::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:34:02,040::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:34:06,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:34:06,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0360631 s, 114 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:34:10,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:34:10,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133995 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:34:16,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:34:16,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197296 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:34:20,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:34:20,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845403 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:34:26,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:34:26,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840358 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:34:30,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:34:30,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743491 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:34:36,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:34:36,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242226 s, 169 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:34:40,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:34:40,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904167 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:34:46,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:34:46,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935886 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:34:50,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:34:50,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717426 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:34:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:34:57,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00374319 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:00,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:00,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204847 s, 200 kB/s\n'; <rc> = 0 >Thread-14227::WARNING::2013-03-17 13:35:02,074::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:35:02,076::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:35:07,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:35:07,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904237 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:10,714::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:35:10,715::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:35:10,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:10,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000685726 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:10,753::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000685726 s, 3.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:35:10,753::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:35:10,753::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:35:10,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:35:10,756::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:35:10,757::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:35:10,757::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:35:10,758::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:35:10,758::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:35:10,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:11,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:11,207::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:35:11,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:35:11,210::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:35:11,211::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:35:11,212::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:35:11,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:11,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:11,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:11,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204553 s, 200 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:35:12,079::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:35:12,080::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:35:12,511::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:35:12,517::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:35:12,519::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:35:17,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:35:17,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0368857 s, 111 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:21,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:21,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108648 s, 377 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:35:27,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:35:27,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169575 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:31,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:31,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773178 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:35:37,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:35:37,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103941 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:41,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:41,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162155 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:35:47,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:35:47,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741979 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:35:51,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:35:51,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167562 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:35:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:35:57,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015017 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:36:01,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:36:02,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207421 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:07,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:07,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162338 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:36:12,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:36:12,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00531826 s, 770 kB/s\n'; <rc> = 0 >Thread-14228::WARNING::2013-03-17 13:36:12,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:36:12,557::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:36:17,692::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:36:17,693::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:36:17,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000847618 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:17,719::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000847618 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:36:17,719::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:36:17,720::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:36:17,721::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:36:17,722::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:17,723::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:17,723::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:17,724::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:17,724::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:17,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:18,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:18,122::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:18,123::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:36:18,126::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:36:18,127::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:36:18,128::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:36:18,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:18,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:18,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:18,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00623003 s, 657 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:36:22,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:36:22,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844817 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:36:22,560::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:36:22,562::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:36:22,975::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:36:22,981::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:36:22,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:36:28,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:28,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165716 s, 247 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:36:32,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:36:32,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168216 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:38,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:38,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134542 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:36:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:36:42,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882454 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:48,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:48,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789303 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:36:52,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:36:52,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872431 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:36:58,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:36:58,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772934 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:37:02,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:37:02,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161766 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:37:08,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:37:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0246097 s, 166 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:37:12,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:37:12,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145093 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:37:19,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:37:19,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141482 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:37:22,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:37:22,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00312851 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14229::WARNING::2013-03-17 13:37:23,011::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:37:23,014::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:37:29,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:37:29,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810595 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:37:32,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:37:32,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092829 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:37:33,016::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:37:33,018::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:37:33,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:37:33,435::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:37:33,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:37:39,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:37:39,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182845 s, 224 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:37:42,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:37:42,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220078 s, 186 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:37:49,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:37:49,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077084 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:37:52,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:37:52,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202052 s, 203 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:37:59,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:37:59,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858443 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:38:02,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:38:02,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074396 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:38:09,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:38:09,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238822 s, 172 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:38:12,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:38:12,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225961 s, 181 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:38:19,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:38:19,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146589 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:38:22,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:38:22,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100027 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:38:29,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:38:29,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795881 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:38:32,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:38:33,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118669 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14230::WARNING::2013-03-17 13:38:33,460::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:38:33,464::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:38:39,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:38:39,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00541404 s, 757 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:38:43,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:38:43,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0322374 s, 127 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:38:43,467::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:38:43,468::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:38:43,947::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:38:43,954::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:38:43,955::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:38:49,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:38:49,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155486 s, 263 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:38:53,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:38:53,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140565 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:38:59,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:38:59,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140156 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:39:03,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:39:03,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138023 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:39:09,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:39:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812418 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:39:13,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:39:13,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804382 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:39:19,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:39:20,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759235 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:39:23,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:39:23,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783335 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:39:30,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:39:30,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138135 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:39:33,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:39:33,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105034 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:39:40,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:39:40,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227643 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:39:43,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:39:43,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0039351 s, 1.0 MB/s\n'; <rc> = 0 >Thread-14231::WARNING::2013-03-17 13:39:43,985::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:39:43,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:39:50,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:39:50,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790695 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:39:53,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:39:53,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076098 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:39:53,991::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:39:53,992::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:39:53,993::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:39:54,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:39:54,465::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:39:54,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:40:00,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:40:00,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221701 s, 185 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:03,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:03,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194868 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:40:10,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:40:10,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885031 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:13,675::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:40:13,677::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:40:13,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:13,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0222564 s, 92.0 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:13,728::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0222564 s, 92.0 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:40:13,729::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:40:13,729::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:40:13,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:40:13,733::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:40:13,734::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:40:13,734::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:40:13,735::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:40:13,735::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:40:13,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:14,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:14,180::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:40:14,181::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:40:14,184::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:40:14,185::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:40:14,186::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:40:14,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:14,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:14,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:14,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175962 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:40:20,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:40:20,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163244 s, 251 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:24,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:24,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014746 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:40:30,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:40:30,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00287274 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:34,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:34,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141163 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:40:40,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:40:40,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851961 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:40:44,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:44,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967829 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:40:50,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:40:50,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802795 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14232::WARNING::2013-03-17 13:40:54,490::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:40:54,495::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:40:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:40:54,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742018 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:00,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:00,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152727 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:41:04,502::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:41:04,503::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:04,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:04,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172446 s, 238 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:41:05,020::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:41:05,025::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:41:05,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:10,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:10,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175714 s, 233 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:41:14,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:14,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156219 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:20,944::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:41:20,946::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:41:20,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:20,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00078348 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:20,978::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00078348 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:41:20,979::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:41:20,979::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:41:20,981::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:41:20,982::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:20,982::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:20,983::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:20,983::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:20,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:20,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:21,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:21,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:41:21,420::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:41:21,422::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:41:21,423::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:41:21,424::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:41:21,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:21,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:21,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:21,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223314 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:41:24,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:24,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137533 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:31,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:31,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149079 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:41:34,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:35,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918098 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:41,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:42,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140667 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:41:45,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:45,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00546226 s, 750 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:41:52,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:41:52,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769745 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:41:55,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:41:55,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895895 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:42:02,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:42:02,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137763 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14233::WARNING::2013-03-17 13:42:05,057::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:42:05,060::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:42:05,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:05,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703128 s, 583 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:42:12,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:42:12,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797687 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:42:15,062::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:42:15,063::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:15,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:15,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00892803 s, 459 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:42:15,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:42:15,618::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:42:15,619::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:42:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:42:22,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239207 s, 171 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:42:25,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:25,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134697 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:42:32,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:42:32,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799952 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:42:35,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:35,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843655 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:42:42,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:42:42,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993575 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:42:45,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:45,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243698 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:42:52,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:42:52,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809964 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:42:55,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:42:55,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205899 s, 199 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:43:02,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:43:02,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157106 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:43:05,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:05,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200852 s, 204 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:43:12,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:43:12,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218112 s, 1.9 MB/s\n'; <rc> = 0 >Thread-14234::WARNING::2013-03-17 13:43:15,660::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:43:15,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:43:15,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:15,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878218 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:43:22,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:43:22,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079369 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:43:25,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:43:25,672::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:25,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:25,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193204 s, 212 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:43:26,144::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:43:26,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:43:26,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:43:32,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:43:32,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143185 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:43:35,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:35,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144233 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:43:42,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:43:42,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100016 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:43:45,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:45,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083952 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:43:53,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:43:53,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170945 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:43:55,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:43:55,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162989 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:44:03,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:44:03,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250917 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:44:05,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:44:05,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025726 s, 159 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:44:13,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:44:13,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140511 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:44:15,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:44:16,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843616 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:44:23,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:44:23,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222391 s, 184 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:44:26,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:44:26,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933373 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14235::WARNING::2013-03-17 13:44:26,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:44:26,206::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:44:33,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:44:33,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888612 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:44:36,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:44:36,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112949 s, 363 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:44:36,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:44:36,214::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:44:36,640::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:44:36,647::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:44:36,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:44:43,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:44:43,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835485 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:44:46,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:44:46,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936646 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:44:53,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:44:53,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807838 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:44:56,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:44:56,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00339409 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:45:03,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:45:03,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263711 s, 155 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:06,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:06,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781869 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:45:13,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:45:13,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797947 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:16,291::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:45:16,292::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:45:16,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:16,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00123154 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:16,336::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00123154 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:45:16,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:45:16,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:45:16,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:45:16,339::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:16,339::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:16,340::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:16,340::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:16,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:16,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:16,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:16,783::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:16,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:45:16,786::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:45:16,788::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:45:16,788::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:45:16,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:17,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:17,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:17,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022849 s, 179 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:45:23,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:45:23,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148081 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:27,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:27,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143788 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:45:33,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:45:33,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134838 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14236::WARNING::2013-03-17 13:45:36,674::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:45:36,680::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:45:37,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:37,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852376 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:45:43,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:45:44,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00894553 s, 458 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:45:46,684::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:45:46,685::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:45:46,686::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:45:47,132::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:45:47,139::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:45:47,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:45:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:47,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141848 s, 289 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:45:54,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:45:54,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111606 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:45:57,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:45:57,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166673 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:04,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:04,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0247969 s, 165 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:46:07,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:46:07,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804461 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:14,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:14,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132633 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:46:17,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:46:17,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879251 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:24,298::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:46:24,300::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:46:24,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:24,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118654 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:24,339::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118654 s, 1.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:46:24,339::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:46:24,339::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:46:24,341::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:46:24,342::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:46:24,343::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:46:24,343::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:46:24,343::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:46:24,344::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:46:24,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:24,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:24,779::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:46:24,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:46:24,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:46:24,784::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:46:24,784::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:46:24,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:25,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:25,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:25,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0412887 s, 99.2 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:46:27,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:46:27,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803714 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:35,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:35,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139934 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:46:37,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:46:37,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014624 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:45,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:45,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111647 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14237::WARNING::2013-03-17 13:46:47,167::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:46:47,171::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:46:47,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:46:47,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867593 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:46:55,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:46:55,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158668 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:46:57,173::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:46:57,175::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:46:57,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 13:46:57,651::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-22::DEBUG::2013-03-17 13:46:57,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136001 s, 301 kB/s\n'; <rc> = 0 >Thread-21::WARNING::2013-03-17 13:46:57,686::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:46:57,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:47:05,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:47:05,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792595 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:47:07,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:47:07,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925051 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:47:15,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:47:15,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192736 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:47:17,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:47:17,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728619 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:47:25,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:47:25,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888319 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:47:27,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:47:27,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142791 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:47:35,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:47:35,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0240233 s, 171 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:47:37,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:47:37,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822995 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:47:45,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:47:45,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131743 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:47:47,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:47:47,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147782 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:47:56,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:47:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937574 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14238::WARNING::2013-03-17 13:47:57,715::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:47:57,718::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:47:57,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:47:57,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849294 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:48:06,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:48:06,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911864 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:48:07,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:48:07,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:07,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:07,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162255 s, 252 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:48:08,196::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:48:08,202::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:48:08,203::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:48:16,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:48:16,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920129 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:48:17,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:18,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093164 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:48:26,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:48:26,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752266 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:48:28,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:28,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190578 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:48:36,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:48:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092512 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:48:38,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:38,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076989 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:48:46,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:48:46,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012829 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:48:48,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:48,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138947 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:48:56,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:48:56,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775943 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:48:58,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:48:58,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879095 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:49:06,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:49:06,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152947 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:49:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-14239::WARNING::2013-03-17 13:49:08,230::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:49:08,257::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:49:08,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108983 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:49:16,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:49:16,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197047 s, 208 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:49:18,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:49:18,263::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:49:18,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:49:18,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963364 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:49:18,691::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:49:18,697::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:49:18,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:49:27,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:49:27,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238842 s, 171 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:49:28,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:49:28,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790686 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:49:37,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:49:37,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816915 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:49:38,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:49:38,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209377 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:49:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:49:47,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143401 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:49:48,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:49:48,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00503679 s, 813 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:49:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:49:57,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814188 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:49:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:49:58,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172259 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:50:07,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:50:07,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153673 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:08,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:08,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855016 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:50:17,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:50:17,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000668062 s, 6.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:18,585::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:50:18,591::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:50:18,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:18,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0206421 s, 99.2 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:18,664::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0206421 s, 99.2 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:50:18,666::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:50:18,668::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:50:18,673::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:50:18,679::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:18,683::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:18,686::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:18,688::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:18,690::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:18,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-14240::WARNING::2013-03-17 13:50:18,735::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:50:18,744::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:50:19,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:19,156::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:19,156::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:50:19,159::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:50:19,160::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:50:19,160::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:50:19,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:19,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:19,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:19,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182494 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:50:27,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:50:27,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174133 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:50:28,748::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:50:28,749::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:50:29,178::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:50:29,185::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:50:29,186::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:50:29,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129895 s, 315 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:50:37,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:50:37,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797326 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:39,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975229 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:50:48,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:50:48,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852909 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:49,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803069 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:50:58,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:50:58,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0264563 s, 155 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:50:59,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:50:59,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782145 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:08,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:08,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224372 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:51:09,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:51:09,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074317 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:18,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:18,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147198 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:51:19,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:51:19,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140205 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:28,533::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:51:28,534::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:51:28,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:28,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000837722 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:28,590::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000837722 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:51:28,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:51:28,594::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:51:28,598::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:51:28,603::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:51:28,607::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:51:28,609::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:51:28,612::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:51:28,614::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:51:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:29,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:29,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:51:29,081::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:51:29,093::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:51:29,096::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:51:29,098::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:51:29,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-14241::WARNING::2013-03-17 13:51:29,215::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:51:29,223::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:51:29,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:29,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:29,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145951 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:51:29,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:51:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00459586 s, 891 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:51:39,226::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:51:39,227::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:51:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:39,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:39,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105601 s, 388 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:51:39,681::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:51:39,687::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:51:39,688::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:51:39,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:51:39,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183148 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:49,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023475 s, 174 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:51:49,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:51:49,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778407 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:51:59,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:51:59,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135053 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:51:59,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:51:59,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699162 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:52:09,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:52:09,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150224 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:52:09,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:52:10,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818353 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:52:19,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:52:19,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081177 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:52:20,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:52:20,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140986 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:52:29,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:52:30,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163368 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:52:30,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:52:30,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813106 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14242::WARNING::2013-03-17 13:52:39,717::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:52:39,721::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:52:40,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:52:40,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901374 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:52:40,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:52:40,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208059 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:52:49,723::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:52:49,724::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:52:50,145::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:52:50,151::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:52:50,153::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:52:50,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:52:50,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:52:50,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207729 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:52:50,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128562 s, 319 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:53:00,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:53:00,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141439 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:53:00,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:53:00,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143639 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:53:10,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:53:10,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861252 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:53:10,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:53:10,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152532 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:53:20,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:53:20,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133256 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:53:20,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:53:20,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154977 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:53:30,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:53:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203504 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:53:30,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:53:30,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864315 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:53:40,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:53:40,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133689 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:53:40,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:53:40,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162023 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14243::WARNING::2013-03-17 13:53:50,179::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:53:50,184::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:53:50,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:53:50,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101712 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:53:50,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:53:50,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848193 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:54:00,186::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:54:00,187::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:54:00,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 13:54:00,631::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:54:00,646::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:54:00,652::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:54:00,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132615 s, 309 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:54:00,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:54:00,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174404 s, 235 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:54:10,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:54:10,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154782 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:54:10,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:54:10,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145968 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:54:20,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:54:20,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000685713 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:54:20,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:54:21,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00835444 s, 490 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:54:30,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:54:30,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719185 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:54:31,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:54:31,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129571 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:54:40,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:54:40,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013421 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:54:41,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:54:41,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157313 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:54:50,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:54:50,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164435 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:54:51,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:54:51,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137466 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14244::WARNING::2013-03-17 13:55:00,714::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:55:00,720::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:55:00,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:00,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0043105 s, 950 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:55:01,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:55:01,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100942 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:55:10,724::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:55:10,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:10,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:11,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136551 s, 300 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:55:11,181::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:55:11,187::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:55:11,188::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:55:11,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:55:11,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111176 s, 368 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:21,043::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 13:55:21,045::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:55:21,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:21,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000796893 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:21,082::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000796893 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 13:55:21,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 13:55:21,083::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 13:55:21,085::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 13:55:21,086::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:55:21,087::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:55:21,088::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:55:21,088::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:55:21,089::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 13:55:21,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:21,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:21,514::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:55:21,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:21,541::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 13:55:21,562::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 13:55:21,563::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 13:55:21,565::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 13:55:21,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 13:55:21,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229276 s, 179 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:22,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:22,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:22,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163128 s, 251 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:55:31,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:55:31,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128433 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:32,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:32,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904245 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:55:41,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:55:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984946 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:42,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:42,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762228 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:55:51,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:55:51,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139111 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:55:52,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:55:52,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110431 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:01,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:01,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159763 s, 256 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:56:02,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:02,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120447 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14245::WARNING::2013-03-17 13:56:11,216::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:56:11,219::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 13:56:12,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:12,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971031 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:56:12,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:12,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775627 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:56:21,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:56:21,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:56:21,637::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:56:21,644::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:56:21,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:22,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:22,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146878 s, 279 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:56:22,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:22,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159862 s, 256 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:32,190::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 13:56:32,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:56:32,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:32,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149054 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:32,238::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149054 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 13:56:32,239::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 13:56:32,239::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 13:56:32,241::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 13:56:32,242::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:32,242::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:32,243::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:32,243::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:32,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:32,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:32,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:32,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138114 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:32,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:32,725::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 13:56:32,726::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 13:56:32,728::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 13:56:32,730::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 13:56:32,730::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 13:56:32,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:33,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:33,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:33,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00736254 s, 556 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:56:42,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:42,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827658 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:43,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:43,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228283 s, 179 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:56:52,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:56:52,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803512 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:56:53,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:56:53,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135524 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:57:02,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:57:02,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176863 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:57:03,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:57:03,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218671 s, 187 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:57:12,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:57:12,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002182 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:57:13,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:57:13,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154599 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14246::WARNING::2013-03-17 13:57:21,670::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:57:21,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:57:22,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:57:22,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206323 s, 199 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:57:23,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:57:23,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193231 s, 212 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:57:31,675::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 13:57:31,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:57:31,678::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:57:32,095::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:57:32,102::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:57:32,103::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:57:32,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:57:32,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182501 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:57:33,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:57:33,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140991 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:57:42,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:57:42,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130302 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:57:43,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:57:43,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176944 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:57:52,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:57:52,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127568 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:57:53,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:57:53,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139746 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:58:02,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:58:02,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001054 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:58:03,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:58:03,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070762 s, 579 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:58:12,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:58:12,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020951 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:58:13,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:58:13,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797511 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:58:22,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:58:22,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888585 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:58:24,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:58:24,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0407384 s, 101 kB/s\n'; <rc> = 0 >Thread-14247::WARNING::2013-03-17 13:58:32,129::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:58:32,133::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:58:32,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:58:33,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819899 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:58:34,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:58:34,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917138 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:58:42,135::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:58:42,136::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:58:42,556::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:58:42,563::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:58:42,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:58:43,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:58:43,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170113 s, 241 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:58:44,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:58:44,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0316105 s, 130 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:58:53,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:58:53,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129532 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:58:54,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:58:54,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156441 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:59:03,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:59:03,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974356 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:59:04,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:59:04,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965754 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:59:13,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:59:13,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00249423 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:59:14,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:59:14,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788313 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:59:23,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:59:23,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126385 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:59:24,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:59:24,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180986 s, 226 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 13:59:33,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:59:33,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138775 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:59:34,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:59:34,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083479 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14248::WARNING::2013-03-17 13:59:42,593::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 13:59:42,598::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 13:59:43,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:59:43,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767074 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:59:44,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:59:44,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787013 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 13:59:52,602::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 13:59:52,603::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 13:59:53,015::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 13:59:53,021::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 13:59:53,023::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 13:59:53,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 13:59:53,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139852 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 13:59:54,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 13:59:54,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112004 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:03,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194046 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:00:04,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:00:04,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018522 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:13,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:13,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765196 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:00:14,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:00:15,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771601 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:23,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:00:23,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:00:23,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:23,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00241791 s, 847 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:23,607::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00241791 s, 847 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:00:23,608::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:00:23,608::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:00:23,609::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:00:23,610::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:00:23,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:24,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:24,022::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:00:24,023::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:00:24,025::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:00:24,026::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:00:24,027::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:00:24,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:24,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:24,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:24,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123144 s, 333 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:00:25,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:00:25,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125572 s, 326 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:34,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:34,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802545 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:00:35,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:00:35,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866101 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:00:44,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:44,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206601 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:00:45,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:00:45,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159355 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14249::WARNING::2013-03-17 14:00:53,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:00:53,053::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:00:54,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:00:54,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825575 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:00:55,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:00:55,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868069 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:01:03,055::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:01:03,056::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:01:03,519::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:01:03,526::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:01:03,528::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:01:04,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:01:04,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638781 s, 641 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:05,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:05,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01716 s, 239 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:01:14,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:01:14,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783828 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:15,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:15,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771694 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:01:24,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:01:24,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186466 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:25,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:25,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016921 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:01:34,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:01:34,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873459 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:35,618::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:01:35,621::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:01:35,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:35,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00100871 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:35,651::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00100871 s, 2.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:01:35,651::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:01:35,652::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:01:35,653::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:01:35,654::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:01:35,655::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:01:35,656::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:01:35,656::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:01:35,656::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:01:35,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:36,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:36,097::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:01:36,098::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:01:36,101::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:01:36,102::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:01:36,103::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:01:36,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:36,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:36,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:36,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00778247 s, 526 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:01:44,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:01:44,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812104 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:46,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:46,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078135 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:01:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:01:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125919 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:01:56,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:01:56,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844416 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14250::WARNING::2013-03-17 14:02:03,559::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:02:03,565::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:02:04,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:02:04,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118887 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:02:06,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:02:06,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828021 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:02:13,576::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:02:13,577::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:02:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:02:14,031::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:02:14,033::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:02:14,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:02:14,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125899 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:02:16,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:02:16,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103878 s, 394 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:02:24,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:02:24,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093506 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:02:26,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:02:26,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163979 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:02:34,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:02:35,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800002 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:02:36,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:02:37,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864716 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:02:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:02:45,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141639 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:02:47,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:02:47,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849267 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:02:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:02:55,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00677288 s, 605 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:02:57,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:02:57,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001486 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:03:05,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:03:05,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145331 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:03:07,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:03:07,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150333 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14251::WARNING::2013-03-17 14:03:14,062::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:03:14,069::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:03:15,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:03:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089147 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:03:17,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:03:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848539 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:03:24,085::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:03:24,086::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:03:24,087::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:03:24,521::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:03:24,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:03:24,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:03:25,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:03:25,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012445 s, 329 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:03:27,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:03:27,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208437 s, 197 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:03:35,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:03:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148416 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:03:37,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:03:37,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950635 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:03:45,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:03:45,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874252 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:03:47,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:03:47,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220577 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:03:55,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:03:55,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00246957 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:03:57,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:03:57,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134537 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:04:05,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:04:05,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132654 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:04:07,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:04:07,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143216 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:04:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:04:15,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135311 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:04:17,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:04:17,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228258 s, 179 kB/s\n'; <rc> = 0 >Thread-14252::WARNING::2013-03-17 14:04:24,553::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:04:24,557::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:04:25,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:04:25,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780968 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:04:27,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:04:27,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0296346 s, 138 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:04:34,559::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:04:34,560::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:04:35,011::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:04:35,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:04:35,019::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:04:35,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:04:35,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0432 s, 94.8 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:04:37,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:04:38,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101652 s, 403 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:04:45,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:04:45,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110082 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:04:48,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:04:48,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092631 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:04:55,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:04:55,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839144 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:04:58,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:04:58,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156019 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:05,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:05,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000675262 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:05:08,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:05:08,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129192 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:15,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:15,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916695 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:05:18,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:05:18,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178955 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:25,700::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:05:25,701::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:05:25,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:25,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000802093 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:25,732::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000802093 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:05:25,733::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:05:25,733::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:05:25,735::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:05:25,743::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:25,744::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:25,744::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:25,745::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:25,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:25,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:26,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:26,193::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:26,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:05:26,196::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:05:26,197::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:05:26,198::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:05:26,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:26,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:26,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:26,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168491 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:05:28,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:05:28,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212322 s, 193 kB/s\n'; <rc> = 0 >Thread-14253::WARNING::2013-03-17 14:05:35,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:05:35,054::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:05:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:36,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832297 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:05:38,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:05:38,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821091 s, 499 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:05:45,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:05:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:05:45,522::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:05:45,529::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:05:45,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:05:46,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:46,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117197 s, 349 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:05:48,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:05:48,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244653 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:05:56,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:05:56,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916373 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:05:58,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:05:58,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877811 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:06:06,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:06:06,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150415 s, 272 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:08,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:08,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151932 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:06:16,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:06:16,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129496 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:18,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:18,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0374417 s, 109 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:06:26,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:06:26,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843987 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:28,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:28,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845743 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:06:36,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:06:36,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740135 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:39,027::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:06:39,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:06:39,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:39,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121206 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:39,072::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121206 s, 1.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:06:39,072::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:06:39,073::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:06:39,074::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:06:39,078::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:06:39,078::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:06:39,079::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:06:39,079::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:06:39,080::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:06:39,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:39,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:39,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:06:39,543::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:06:39,548::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:06:39,549::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:06:39,549::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:06:39,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:39,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:39,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:39,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113824 s, 360 kB/s\n'; <rc> = 0 >Thread-14254::WARNING::2013-03-17 14:06:45,558::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:06:45,565::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:06:46,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:06:46,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895032 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:06:49,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:06:50,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964693 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:06:55,567::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:06:55,568::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:06:56,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:06:56,076::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:06:56,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:06:57,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:06:57,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139863 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:07:00,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:07:00,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179323 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:07:07,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:07:07,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781261 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:07:10,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:07:10,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155604 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:07:17,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:07:17,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145521 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:07:20,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:07:20,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846574 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:07:27,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:07:27,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168412 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:07:30,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:07:30,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892649 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:07:37,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:07:37,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155652 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:07:40,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:07:40,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090081 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:07:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:07:47,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142639 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:07:50,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:07:50,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894761 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14255::WARNING::2013-03-17 14:07:56,105::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:07:56,112::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:07:57,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:07:57,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209054 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:08:00,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:08:00,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840955 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:08:06,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:08:06,116::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:08:06,557::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:08:06,567::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:08:06,568::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:08:07,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:08:07,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146308 s, 280 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:08:10,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:08:10,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199428 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:08:17,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:08:17,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840207 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:08:20,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:08:20,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267161 s, 153 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:08:27,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:08:27,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851884 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:08:30,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:08:30,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084043 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:08:37,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:08:37,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156717 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:08:40,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:08:40,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267337 s, 153 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:08:47,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:08:47,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144015 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:08:50,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:08:51,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140031 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:08:57,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:08:57,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170066 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:09:01,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:09:01,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146163 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14256::WARNING::2013-03-17 14:09:06,597::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:09:06,601::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:09:07,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:09:07,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870225 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:09:11,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:09:11,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889241 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:09:16,602::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:09:16,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:09:16,605::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:09:17,052::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:09:17,059::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:09:17,060::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:09:17,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:09:17,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108734 s, 377 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:09:21,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:09:21,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175407 s, 234 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:09:27,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:09:27,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165099 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:09:31,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:09:31,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206137 s, 199 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:09:37,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:09:37,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166425 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:09:41,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:09:41,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142124 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:09:47,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:09:47,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157566 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:09:51,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:09:51,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0274733 s, 149 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:09:57,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:09:57,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868354 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:10:01,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:10:01,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016531 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:07,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:07,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830087 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:10:11,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:10:11,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013681 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14257::WARNING::2013-03-17 14:10:17,091::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:10:17,096::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:10:17,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:17,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912589 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:10:21,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:10:21,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750698 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:10:27,101::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:10:27,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:10:27,565::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:10:27,571::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:10:27,572::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:27,932::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:10:27,935::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:10:27,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:27,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0185713 s, 110 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:27,985::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0185713 s, 110 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:10:27,985::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:10:27,985::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:10:27,987::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:10:27,988::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:27,989::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:27,989::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:27,989::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:27,990::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:27,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:28,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:28,360::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:10:28,361::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:10:28,364::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:10:28,365::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:10:28,365::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:10:28,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:28,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:28,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:28,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125953 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:10:31,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:10:32,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157019 s, 261 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:38,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:38,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231683 s, 177 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:10:42,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:10:42,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827991 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:48,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:48,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069244 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:10:52,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:10:52,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013726 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:10:58,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:10:58,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774308 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:02,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:02,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151236 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:11:08,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:11:08,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139112 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:12,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:12,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131048 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:11:18,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:11:18,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135658 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:22,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:22,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125639 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14258::WARNING::2013-03-17 14:11:27,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:11:27,611::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:11:28,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:11:29,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689885 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:32,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:32,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839209 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:11:37,613::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:11:37,614::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:11:38,096::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:11:38,103::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:11:38,104::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:11:39,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:11:39,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193206 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:42,504::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:11:42,509::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:11:42,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:42,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0149951 s, 137 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:42,553::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0149951 s, 137 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:11:42,553::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:11:42,554::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:11:42,555::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:11:42,556::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:11:42,557::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:11:42,557::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:11:42,558::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:11:42,558::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:11:42,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:43,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:43,005::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:11:43,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:11:43,010::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:11:43,011::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:11:43,011::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:11:43,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:43,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:43,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:43,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136032 s, 301 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:11:49,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:11:49,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774102 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:11:53,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:11:53,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770824 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:11:59,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:11:59,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196524 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:12:03,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:12:03,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833248 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:12:09,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:12:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219014 s, 187 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:12:13,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:12:13,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834201 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:12:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:12:19,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783333 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:12:23,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:12:23,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103644 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:12:29,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:12:29,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836031 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:12:33,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:12:33,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163988 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14259::WARNING::2013-03-17 14:12:38,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:12:38,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:12:39,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:12:39,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833471 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:12:43,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:12:43,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096014 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:12:48,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:12:48,139::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:12:48,593::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:12:48,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:12:48,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:12:49,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:12:49,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010089 s, 406 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:12:53,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:12:53,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177253 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:12:59,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:12:59,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142445 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:13:04,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:13:04,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119194 s, 344 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:13:09,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:13:09,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077334 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:13:14,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:13:14,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157867 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:13:19,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:13:19,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137836 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:13:24,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:13:24,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112318 s, 365 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:13:29,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:13:29,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009837 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:13:34,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:13:34,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872646 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:13:39,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:13:39,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773438 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:13:44,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:13:44,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723693 s, 5.7 MB/s\n'; <rc> = 0 >Thread-14260::WARNING::2013-03-17 14:13:48,630::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:13:48,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:13:49,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:13:49,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985959 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:13:54,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:13:54,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076635 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:13:58,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:13:58,641::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:13:59,088::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:13:59,096::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:13:59,097::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:13:59,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:13:59,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0300149 s, 136 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:14:04,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:14:04,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171593 s, 239 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:14:09,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:14:09,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014146 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:14:14,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:14:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853405 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:14:19,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:14:19,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108902 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:14:24,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:14:24,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855257 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:14:29,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:14:29,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179874 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:14:34,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:14:34,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01292 s, 317 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:14:39,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:14:39,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00618519 s, 662 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:14:44,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:14:44,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876274 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:14:49,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:14:49,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150345 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:14:54,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:14:54,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814134 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14261::WARNING::2013-03-17 14:14:59,130::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:14:59,135::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:14:59,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:00,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182312 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:15:04,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:15:05,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901776 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:15:09,137::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:15:09,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:15:09,141::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:15:09,593::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:15:09,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:15:09,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:10,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:10,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165088 s, 248 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:15:15,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:15:15,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0307499 s, 133 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:20,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:20,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184294 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:15:25,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:15:25,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747706 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:30,143::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:15:30,145::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:15:30,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:30,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00123209 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:30,187::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00123209 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:15:30,187::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:15:30,188::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:15:30,189::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:15:30,190::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:30,191::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:30,191::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:30,192::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:30,192::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:30,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:30,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:30,614::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:15:30,615::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:15:30,617::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:15:30,618::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:15:30,619::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:15:30,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:30,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:30,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:31,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177825 s, 230 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:15:35,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:15:35,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112194 s, 365 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:41,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:41,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076532 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:15:45,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:15:45,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897922 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:15:51,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:15:51,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934676 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:15:55,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:15:55,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150424 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:16:01,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:16:01,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148803 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:05,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:05,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132094 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14262::WARNING::2013-03-17 14:16:09,637::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:16:09,643::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:16:11,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:16:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902211 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:15,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:15,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757116 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:16:19,646::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:16:19,647::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:16:20,087::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:16:20,093::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:16:20,095::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:16:21,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:16:21,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184586 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:25,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:25,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119173 s, 344 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:16:31,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:16:31,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00350368 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:35,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:35,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939981 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:16:41,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:16:41,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144147 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:45,769::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:16:45,771::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:16:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:45,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000863674 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:45,800::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000863674 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:16:45,800::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:16:45,801::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:16:45,802::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:16:45,803::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:16:45,804::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:16:45,805::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:16:45,805::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:16:45,805::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:16:45,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:46,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:46,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:16:46,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:16:46,241::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:16:46,243::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:16:46,243::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:16:46,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:46,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:46,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:46,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968625 s, 423 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:16:51,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:16:51,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859358 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:16:56,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:16:56,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171189 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:17:01,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:17:01,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891757 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:17:06,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:17:06,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132522 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:17:11,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:17:11,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777472 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:17:16,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:17:16,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920849 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14263::WARNING::2013-03-17 14:17:20,120::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:17:20,124::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:17:21,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:17:21,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829125 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:17:26,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:17:26,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781334 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:17:30,127::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:17:30,128::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:17:30,567::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:17:30,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:17:30,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:17:31,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:17:31,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140308 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:17:37,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:17:37,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128055 s, 320 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:17:41,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:17:41,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118448 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:17:47,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:17:47,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153161 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:17:51,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:17:51,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873163 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:17:57,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:17:57,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116274 s, 352 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:18:01,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:18:01,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764405 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:18:07,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:18:07,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136377 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:18:11,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:18:11,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199606 s, 205 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:18:17,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:18:17,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146316 s, 280 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:18:21,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:18:21,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217445 s, 188 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:18:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:18:27,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791868 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14264::WARNING::2013-03-17 14:18:30,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:18:30,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:18:31,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:18:31,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860326 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:18:37,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:18:37,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241661 s, 169 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:18:40,606::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:18:40,607::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:18:41,104::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:18:41,111::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:18:41,113::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:18:41,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:18:41,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125603 s, 326 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:18:47,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:18:47,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109082 s, 375 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:18:51,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:18:51,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100362 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:18:57,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:18:57,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240903 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:19:01,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:19:01,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881018 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:19:07,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:19:07,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193805 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:19:11,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:19:11,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153565 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:19:17,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:19:17,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163964 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:19:21,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:19:22,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843523 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:19:27,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:19:27,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076333 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:19:32,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:19:32,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075383 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:19:38,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:19:38,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155987 s, 263 kB/s\n'; <rc> = 0 >Thread-14265::WARNING::2013-03-17 14:19:41,159::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:19:41,164::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:19:42,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:19:42,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709154 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:19:48,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:19:48,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766761 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:19:51,168::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:19:51,170::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:19:51,608::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:19:51,615::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:19:51,616::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:19:52,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:19:52,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019051 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:19:58,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:19:58,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187133 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:02,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:02,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139067 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:20:08,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:20:08,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153188 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:12,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:12,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785318 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:20:18,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:20:18,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023596 s, 174 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:22,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:22,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133527 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:20:28,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:20:28,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873808 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:32,327::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:20:32,328::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:20:32,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:32,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000908734 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:32,359::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000908734 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:20:32,359::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:20:32,359::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:20:32,361::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:20:32,362::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:20:32,363::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:20:32,363::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:20:32,363::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:20:32,364::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:20:32,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:32,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:32,779::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:20:32,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:20:32,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:20:32,784::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:20:32,785::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:20:32,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:33,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:33,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:33,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152685 s, 268 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:20:38,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:20:38,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865641 s, 473 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:20:43,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:43,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855522 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:20:48,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:20:48,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795005 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14266::WARNING::2013-03-17 14:20:51,644::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:20:51,647::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:20:53,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:20:53,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216048 s, 190 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:20:58,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:20:58,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144686 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:21:01,648::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:21:01,650::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:21:01,651::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:21:02,140::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:21:02,146::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:21:02,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:21:03,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:21:03,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160753 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:08,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:08,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174401 s, 235 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:21:13,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:21:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764322 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:18,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160271 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:21:23,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:21:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193065 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:28,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:29,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135338 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:21:33,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:21:33,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281413 s, 146 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:39,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:39,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161494 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:21:43,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:21:43,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014076 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:49,145::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:21:49,146::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:21:49,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:49,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000825198 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:49,178::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000825198 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:21:49,178::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:21:49,179::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:21:49,181::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:21:49,182::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:21:49,183::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:21:49,184::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:21:49,184::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:21:49,184::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:21:49,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:49,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:49,628::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:21:49,629::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:21:49,632::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:21:49,633::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:21:49,633::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:21:49,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:50,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:21:50,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:21:50,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241545 s, 170 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:21:53,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:21:53,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134864 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:22:00,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:22:00,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740462 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14267::WARNING::2013-03-17 14:22:02,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:22:02,194::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:22:03,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:22:03,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869809 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:22:10,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:22:10,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862084 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:22:12,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:22:12,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:22:12,656::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:22:12,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:22:12,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:22:13,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:22:13,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135336 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:22:20,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:22:20,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00801485 s, 511 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:22:23,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:22:23,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205068 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:22:30,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:22:30,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797579 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:22:33,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:22:33,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159133 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:22:40,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:22:40,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219714 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:22:43,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:22:43,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142228 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:22:50,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:22:50,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170226 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:22:53,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:22:53,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900706 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:23:00,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:23:00,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152318 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:23:03,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:23:03,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134983 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:23:10,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:23:10,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231955 s, 177 kB/s\n'; <rc> = 0 >Thread-14268::WARNING::2013-03-17 14:23:12,693::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:23:12,697::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:23:13,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:23:13,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133097 s, 308 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:23:20,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:23:20,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218147 s, 188 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:23:22,699::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:23:22,701::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:23:23,135::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:23:23,142::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:23:23,144::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:23:24,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:23:24,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.038931 s, 105 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:23:30,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:23:30,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173794 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:23:34,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:23:34,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785193 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:23:41,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:23:41,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874143 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:23:44,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:23:44,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181774 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:23:51,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:23:51,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023505 s, 174 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:23:54,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:23:54,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079244 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:24:01,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:24:01,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717576 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:24:04,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:24:04,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108107 s, 379 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:24:11,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:24:11,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139567 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:24:14,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:24:14,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783208 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:24:21,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:24:21,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132082 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14269::WARNING::2013-03-17 14:24:23,193::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:24:23,197::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:24:24,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:24:24,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740315 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:24:31,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:24:31,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839189 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:24:33,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:24:33,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:24:33,792::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:24:33,799::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:24:33,801::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:24:34,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:24:34,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133847 s, 306 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:24:41,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:24:41,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162961 s, 251 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:24:44,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:24:44,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758554 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:24:51,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:24:51,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785854 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:24:54,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:24:54,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932257 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:25:01,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:25:01,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236832 s, 173 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:04,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:04,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103741 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:25:11,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:25:11,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252399 s, 162 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:14,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:14,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153513 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:25:21,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:25:21,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768323 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:24,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:24,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828006 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:25:31,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:25:32,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119983 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14270::WARNING::2013-03-17 14:25:33,827::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:25:33,831::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:25:34,584::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:25:34,586::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:25:34,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:34,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00077613 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:34,615::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00077613 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:25:34,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:25:34,616::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:25:34,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:25:34,619::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:34,620::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:34,621::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:34,621::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:34,622::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:34,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:35,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:35,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:35,073::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:25:35,075::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:25:35,077::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:25:35,077::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:25:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:35,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:35,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:35,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142766 s, 287 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:25:42,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:25:42,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183378 s, 223 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:25:43,833::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:25:43,834::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:25:44,384::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:25:44,391::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:25:44,392::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:25:45,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:45,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0454649 s, 90.1 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:25:52,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:25:52,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215136 s, 190 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:25:55,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:25:55,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019784 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:02,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:02,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191237 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:26:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:26:05,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186907 s, 219 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:12,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:12,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144006 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:26:15,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:26:15,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149079 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:22,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:22,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0246934 s, 166 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:26:25,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:26:25,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127936 s, 320 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:32,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:32,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174176 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:26:35,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:26:35,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854214 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:42,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119625 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14271::WARNING::2013-03-17 14:26:44,417::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:26:44,423::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:26:45,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:26:45,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998389 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:52,699::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:26:52,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:26:52,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:52,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000904246 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:52,728::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000904246 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:26:52,729::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:26:52,729::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:26:52,731::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:26:52,732::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:26:52,732::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:26:52,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:26:52,733::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:26:52,734::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:26:52,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:53,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:53,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:26:53,140::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:26:53,143::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:26:53,144::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:26:53,144::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:26:53,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:53,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:26:53,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:26:53,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285703 s, 1.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:26:54,427::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:26:54,428::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:26:54,429::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:26:54,933::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:26:54,940::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:26:54,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:26:55,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:26:55,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104553 s, 392 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:27:03,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:27:03,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188323 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:27:05,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:27:05,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092599 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:27:13,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:27:13,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016408 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:27:15,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:27:15,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823218 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:27:23,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:27:23,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220215 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:27:26,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:27:26,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845594 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:27:33,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:27:33,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135939 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:27:36,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:27:36,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749698 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:27:43,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:27:43,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906806 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:27:46,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:27:46,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145351 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:27:54,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:27:54,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0260168 s, 157 kB/s\n'; <rc> = 0 >Thread-14272::WARNING::2013-03-17 14:27:54,968::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:27:54,972::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:27:56,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:27:56,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117278 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:28:04,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:28:04,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918848 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:28:04,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:28:04,979::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:28:05,526::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:28:05,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:28:05,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:28:06,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:28:06,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020201 s, 203 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:28:14,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:28:14,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200904 s, 204 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:28:16,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:28:16,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150704 s, 272 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:28:24,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:28:24,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077757 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:28:26,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:28:26,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853389 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:28:34,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:28:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820569 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:28:36,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:28:36,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940716 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:28:44,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:28:44,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731309 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:28:46,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:28:46,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174636 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:28:54,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:28:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822542 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:28:56,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:28:56,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120919 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:29:04,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:05,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846952 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14273::WARNING::2013-03-17 14:29:05,576::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:29:05,580::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:29:06,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:29:06,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999166 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:29:15,583::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:29:15,584::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:15,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:15,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995951 s, 411 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:29:16,116::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:29:16,122::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:29:16,123::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:29:16,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:29:16,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166374 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:29:25,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:25,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865907 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:29:26,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:29:26,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808841 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:29:35,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:35,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096699 s, 424 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:29:36,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:29:36,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974041 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:29:45,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:45,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817912 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:29:46,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:29:46,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780221 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:29:55,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:29:56,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142819 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:29:56,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:29:56,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940682 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:30:06,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:06,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216853 s, 189 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:06,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285135 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14274::WARNING::2013-03-17 14:30:16,158::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:30:16,163::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 14:30:16,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:16,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870883 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:16,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:16,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806225 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:30:26,166::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:30:26,168::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:26,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:26,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0290076 s, 141 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:30:26,687::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:30:26,693::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:30:26,694::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:26,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:26,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182423 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:30:36,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:36,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761318 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:36,928::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:30:36,930::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:30:36,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:36,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000730486 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:36,969::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000730486 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:30:36,969::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:30:36,969::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:30:36,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:30:36,972::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:36,973::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:36,973::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:36,974::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:36,974::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:36,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:37,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:37,410::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:30:37,411::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:30:37,414::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:30:37,415::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:30:37,415::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:30:37,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:37,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:37,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:37,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206508 s, 198 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:30:46,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:46,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019635 s, 209 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:47,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:47,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141627 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:30:56,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:30:56,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135794 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:30:57,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:30:57,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828804 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:06,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:06,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142685 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:31:07,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:31:07,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765904 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:16,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:16,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016627 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:31:17,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:31:18,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162076 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14275::WARNING::2013-03-17 14:31:26,723::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:31:26,725::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 14:31:26,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:26,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163858 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:31:28,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:31:28,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861689 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:31:36,728::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:31:36,730::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:36,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:36,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125766 s, 326 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:31:37,257::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:31:37,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:31:37,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:31:38,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:31:38,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121174 s, 338 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:47,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:47,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827769 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:31:48,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:31:48,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173568 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:57,094::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:31:57,096::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:31:57,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:57,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0233727 s, 87.6 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:57,151::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0233727 s, 87.6 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:31:57,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:31:57,152::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:31:57,154::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:31:57,155::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:31:57,156::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:31:57,157::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:31:57,157::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:31:57,158::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:31:57,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:57,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:57,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:31:57,583::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:31:57,586::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:31:57,587::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:31:57,587::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:31:57,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:57,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:31:57,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:31:58,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151437 s, 270 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:31:58,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:31:58,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145727 s, 281 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:32:08,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:08,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785358 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:32:08,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:32:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823037 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:32:18,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:18,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841518 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:32:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:32:18,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151022 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:32:28,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:28,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707433 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:32:28,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:32:28,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904585 s, 4.5 MB/s\n'; <rc> = 0 >Thread-14276::WARNING::2013-03-17 14:32:37,290::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:32:37,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:32:38,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:38,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:32:38,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129955 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:32:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130191 s, 3.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:32:47,295::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:32:47,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:32:47,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:32:47,894::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:32:47,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:32:47,902::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:32:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:48,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:48,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766811 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:32:48,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173196 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:32:58,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:58,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:32:58,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228464 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:32:58,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204235 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:08,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:08,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:08,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012098 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:08,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115626 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:18,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:18,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:19,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111194 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:19,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205497 s, 199 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:29,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:29,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:29,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888421 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:29,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126942 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:39,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:39,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:39,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112007 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:33:39,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00706329 s, 580 kB/s\n'; <rc> = 0 >Thread-14277::WARNING::2013-03-17 14:33:47,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:33:47,936::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:33:49,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:49,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:33:49,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822044 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:33:49,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138495 s, 3.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:33:57,939::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:33:57,940::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:33:58,403::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:33:58,409::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:33:58,411::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:33:59,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:33:59,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:33:59,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194036 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:33:59,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121591 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:34:09,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:34:09,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085224 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:34:09,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:34:09,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138145 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:34:19,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:34:19,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956211 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:34:19,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:34:19,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846639 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:34:29,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:34:29,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852392 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:34:29,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:34:29,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836537 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:34:39,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:34:39,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000675427 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:34:39,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:34:39,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140961 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:34:49,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:34:49,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204594 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:34:49,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:34:49,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018098 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14278::WARNING::2013-03-17 14:34:58,438::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:34:58,444::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:34:59,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:34:59,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843654 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:35:00,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:00,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852279 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:35:08,446::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:35:08,447::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:35:08,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:35:08,930::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:35:08,931::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:09,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:10,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257105 s, 159 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:35:10,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:10,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785422 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:20,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:20,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070998 s, 577 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:35:20,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:20,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836241 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:30,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:30,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081799 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:35:30,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:30,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982797 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:40,167::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:35:40,168::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:35:40,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:40,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000731388 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:40,195::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000731388 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:35:40,195::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:35:40,195::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:35:40,197::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:35:40,198::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:40,199::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:40,199::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:40,200::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:40,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:40,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:40,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:40,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159685 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:40,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:40,675::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:35:40,676::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:35:40,678::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:35:40,679::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:35:40,680::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:35:40,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:41,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:41,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:41,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906696 s, 452 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:35:50,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:35:50,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145824 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:35:51,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:35:51,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893611 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:36:00,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:36:00,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178584 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:36:01,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:36:01,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278239 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14279::WARNING::2013-03-17 14:36:08,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:36:08,977::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 14:36:10,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:36:10,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000679935 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:36:11,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:36:11,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994312 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:36:18,979::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:36:18,981::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:36:19,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:36:19,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:36:19,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:36:20,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:36:20,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891209 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:36:21,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:36:21,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195187 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:36:30,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:36:30,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868133 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:36:31,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:36:31,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196275 s, 209 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:36:40,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:36:40,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084015 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:36:41,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:36:41,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210146 s, 195 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:36:50,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:36:50,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148255 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:36:51,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:36:51,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174684 s, 234 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:00,937::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:37:00,938::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:37:00,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:00,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000867281 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:00,972::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000867281 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:37:00,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:37:00,973::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:37:00,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:37:00,976::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:37:00,977::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:37:00,978::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:37:00,978::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:37:00,980::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:37:00,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:01,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:01,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:37:01,418::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:37:01,422::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:37:01,423::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:37:01,424::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:37:01,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:01,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:01,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00657208 s, 623 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:01,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:01,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:01,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014025 s, 292 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:37:11,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:11,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00508287 s, 806 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:12,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:12,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160898 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14280::WARNING::2013-03-17 14:37:19,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:37:19,455::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:37:21,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:21,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197847 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:22,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:22,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0329921 s, 124 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:37:29,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:37:29,458::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:37:29,947::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:37:29,954::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:37:29,955::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:37:31,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:31,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201831 s, 203 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:32,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:32,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0261911 s, 156 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:37:41,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:41,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020441 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:42,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140712 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:37:51,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:37:51,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170109 s, 241 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:37:52,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:37:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768623 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:38:01,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:38:01,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190235 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:38:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:38:02,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152195 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:38:11,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:38:11,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173387 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:38:12,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:38:12,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149474 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:38:21,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:38:21,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00892255 s, 459 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:38:22,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:38:22,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147734 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14281::WARNING::2013-03-17 14:38:29,979::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:38:29,983::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:38:31,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:38:31,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917906 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:38:32,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:38:32,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181769 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:38:39,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:38:39,986::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:38:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:38:40,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:38:40,435::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:38:40,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:38:42,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:38:42,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171452 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:38:42,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:38:42,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793267 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:38:52,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:38:52,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880134 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:38:52,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:38:52,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084901 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:39:02,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:39:02,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102028 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:39:02,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:39:02,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243784 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:39:12,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:39:12,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721025 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:39:12,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:39:13,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146948 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:39:22,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:39:22,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190248 s, 215 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:39:23,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:39:23,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157489 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:39:32,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:39:32,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878607 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:39:33,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:39:33,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141979 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14282::WARNING::2013-03-17 14:39:40,464::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:39:40,469::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:39:42,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:39:42,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946969 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:39:43,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:39:43,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808693 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:39:50,473::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:39:50,474::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:39:50,916::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:39:50,923::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:39:50,924::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:39:52,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:39:52,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131513 s, 311 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:39:53,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:39:53,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015432 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:02,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:02,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169072 s, 242 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:40:03,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:40:03,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091234 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:12,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:12,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832418 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:40:13,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:40:13,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162262 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:22,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:22,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852841 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:40:23,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:40:23,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153305 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:32,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:32,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858613 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:40:33,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:40:33,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155652 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:42,514::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:40:42,516::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:40:42,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:42,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.001399 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:42,563::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.001399 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:40:42,563::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:40:42,563::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:40:42,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:40:42,566::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:40:42,567::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:40:42,567::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:40:42,567::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:40:42,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:40:42,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:42,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:43,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:40:43,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:40:43,006::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:40:43,008::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:40:43,008::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:40:43,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:43,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:40:43,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174463 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:40:43,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:40:43,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014984 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14283::WARNING::2013-03-17 14:40:50,964::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:40:50,969::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:40:53,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:40:53,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155362 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:40:53,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:40:53,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160849 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:41:00,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:41:00,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:41:01,451::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:41:01,458::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:41:01,460::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:41:03,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:41:03,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155703 s, 263 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:41:03,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:41:03,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790879 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:41:13,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:41:13,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173055 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:41:13,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:41:13,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152525 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:41:23,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:41:23,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014429 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:41:24,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:41:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147569 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:41:33,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:41:33,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162067 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:41:34,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:41:34,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832075 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:41:43,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:41:43,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000694979 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:41:44,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:41:44,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153599 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:41:53,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:41:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155144 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:41:54,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:41:54,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764018 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14284::WARNING::2013-03-17 14:42:01,487::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:42:01,493::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:42:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:42:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183717 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:04,343::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:42:04,345::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:42:04,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:04,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00096893 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:04,373::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00096893 s, 2.1 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:42:04,374::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:42:04,374::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:42:04,376::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:42:04,377::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:42:04,377::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:42:04,378::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:42:04,378::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:42:04,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:42:04,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:04,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:04,808::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:42:04,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:42:04,811::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:42:04,813::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:42:04,813::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:42:04,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:05,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:05,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:05,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217852 s, 188 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:42:11,496::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:42:11,497::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:42:11,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:42:11,987::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:42:11,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:42:13,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:42:13,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126504 s, 324 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:15,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:15,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746611 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:42:23,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:42:23,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842159 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:25,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763737 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:42:33,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:42:33,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156448 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:35,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:35,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010024 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:42:43,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:42:44,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134427 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:45,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:45,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00480571 s, 852 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:42:54,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:42:54,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141062 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:42:55,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:42:55,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829033 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:43:04,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:43:04,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166013 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:43:05,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:43:05,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239122 s, 171 kB/s\n'; <rc> = 0 >Thread-14285::WARNING::2013-03-17 14:43:12,015::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:43:12,027::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:43:14,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:43:14,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759596 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:43:15,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:43:15,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02234 s, 183 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:43:22,031::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:43:22,032::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:43:22,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:43:22,488::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:43:22,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:43:24,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:43:24,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147858 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:43:25,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:43:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162854 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:43:34,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:43:34,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820353 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:43:35,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:43:35,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866296 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:43:44,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:43:44,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120562 s, 340 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:43:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:43:46,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852734 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:43:54,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:43:54,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115016 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:43:56,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:43:56,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835517 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:44:04,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:44:04,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892158 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:44:06,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:44:06,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135874 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:44:14,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:44:14,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744782 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:44:16,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:44:16,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782809 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14286::WARNING::2013-03-17 14:44:22,517::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:44:22,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:44:24,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:44:24,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184972 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:44:26,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:44:26,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711336 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:44:32,522::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:44:32,524::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:44:32,525::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:44:32,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:44:32,987::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:44:32,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:44:34,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:44:34,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130402 s, 314 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:44:36,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:44:36,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151863 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:44:44,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:44:44,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802314 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:44:46,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:44:46,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909794 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:44:54,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:44:54,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164733 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:44:56,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:44:56,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184699 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:04,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:04,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013318 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:45:06,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:45:06,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142769 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:14,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:14,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226242 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:45:16,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:45:16,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148001 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:24,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:24,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157414 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:45:26,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:45:26,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102295 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14287::WARNING::2013-03-17 14:45:33,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:45:33,022::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:45:34,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:34,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858613 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:45:36,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:45:36,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938034 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:45:43,025::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:45:43,026::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:45:43,468::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:45:43,474::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:45:43,476::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:44,836::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:45:44,845::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:45:44,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:44,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0125259 s, 164 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:44,893::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0125259 s, 164 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:45:44,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:45:44,894::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:45:44,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:45:44,898::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:44,898::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:44,899::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:44,899::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:44,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:44,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:45,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:45,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:45:45,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:45:45,353::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:45:45,354::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:45:45,354::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:45:45,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:45,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:45,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:45,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126967 s, 323 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:45:46,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:45:46,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759777 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:45:55,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:45:55,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817345 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:45:56,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:45:57,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713455 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:46:05,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:46:05,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168266 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:46:07,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:46:07,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244773 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:46:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:46:15,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097914 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:46:17,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:46:17,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834007 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:46:25,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:46:25,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834562 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:46:27,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:46:27,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718315 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:46:35,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:46:35,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793757 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:46:37,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:46:37,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142833 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14288::WARNING::2013-03-17 14:46:43,501::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:46:43,506::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:46:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:46:46,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823973 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:46:47,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:46:47,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094447 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:46:53,508::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:46:53,509::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:46:53,944::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:46:53,950::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:46:53,952::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:46:56,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:46:56,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019523 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:46:57,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:46:57,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772643 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:47:06,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:47:06,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961647 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:07,490::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:47:07,492::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:47:07,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:07,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000763973 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:07,519::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000763973 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:47:07,520::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:47:07,520::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:47:07,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:47:07,523::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:47:07,523::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:47:07,524::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:47:07,524::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:47:07,525::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:47:07,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:07,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:07,943::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:47:07,944::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:47:07,946::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:47:07,947::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:47:07,948::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:47:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:08,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:08,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:08,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122323 s, 335 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:47:16,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:47:16,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798254 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:18,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:18,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788617 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:47:26,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:47:26,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138471 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:28,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:28,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146222 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:47:36,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:47:36,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807764 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:38,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:38,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109399 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:47:46,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:47:46,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162994 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:48,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:48,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879177 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14289::WARNING::2013-03-17 14:47:53,981::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:47:53,984::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:47:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:47:56,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155807 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:47:58,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:47:58,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795734 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:48:03,986::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:48:03,988::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:48:04,431::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:48:04,438::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:48:04,439::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:48:06,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:48:06,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155488 s, 263 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:48:08,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:48:08,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749998 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:48:16,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:48:16,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085919 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:48:18,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:48:18,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160253 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:48:26,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:48:26,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783526 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:48:28,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:48:28,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986559 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:48:36,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:48:36,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946449 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:48:38,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:48:39,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180725 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:48:46,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:48:46,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739111 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:48:49,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:48:49,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119207 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:48:56,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:48:56,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148337 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:48:59,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:48:59,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103147 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14290::WARNING::2013-03-17 14:49:04,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:49:04,472::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:49:06,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:49:06,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740984 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:49:09,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:49:09,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851701 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:49:14,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:49:14,476::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:49:14,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:49:14,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:49:14,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:49:16,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:49:16,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190033 s, 216 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:49:19,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:49:19,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768968 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:49:26,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:49:26,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098858 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:49:29,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:49:29,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840595 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:49:36,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:49:36,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808225 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:49:39,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:49:39,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022152 s, 185 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:49:46,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:49:46,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126833 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:49:49,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:49:49,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789203 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:49:56,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:49:56,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175496 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:49:59,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:49:59,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184824 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:06,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:06,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0031771 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:50:09,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:50:09,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186242 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14291::WARNING::2013-03-17 14:50:14,947::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:50:14,951::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:50:16,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:16,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809519 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:50:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:50:19,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836168 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:50:24,953::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:50:24,954::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:50:24,955::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:50:25,411::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:50:25,418::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:50:25,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:26,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:26,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161844 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:50:29,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:50:29,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449989 s, 910 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:36,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:37,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875574 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:50:39,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:50:39,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137749 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:47,014::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:50:47,015::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:50:47,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:47,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130145 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:47,057::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130145 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:50:47,057::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:50:47,058::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:50:47,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:50:47,060::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:47,062::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:47,062::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:47,063::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:47,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:47,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:47,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:47,496::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:50:47,498::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:50:47,501::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:50:47,503::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:50:47,503::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:50:47,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:47,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:47,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:47,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019237 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:50:49,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:50:50,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857847 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:50:57,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:50:57,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842023 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:51:00,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:51:00,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148686 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:51:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:51:07,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120739 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:51:10,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:51:10,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752041 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:51:17,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:51:18,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818117 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:51:20,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:51:20,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206111 s, 2.0 MB/s\n'; <rc> = 0 >Thread-14292::WARNING::2013-03-17 14:51:25,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:51:25,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:51:28,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:51:28,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904047 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:51:30,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:51:30,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175573 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:51:35,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:51:35,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:51:35,986::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:51:35,993::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:51:35,994::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:51:38,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:51:38,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01544 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:51:40,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:51:40,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857266 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:51:48,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:51:48,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777324 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:51:50,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:51:50,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082874 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:51:58,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:51:58,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872617 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:00,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:00,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143673 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:52:08,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:52:08,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137018 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:10,673::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:52:10,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:52:10,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:10,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0230495 s, 88.9 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:10,740::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0230495 s, 88.9 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:52:10,741::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:52:10,741::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:52:10,743::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:52:10,744::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:52:10,745::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:52:10,745::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:52:10,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:52:10,746::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:52:10,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:11,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:11,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:52:11,186::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:52:11,189::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:52:11,191::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:52:11,192::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:52:11,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:11,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:11,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:11,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00512697 s, 799 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:52:18,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:52:18,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827531 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:21,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:21,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231749 s, 177 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:52:28,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:52:28,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138901 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:31,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:31,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188778 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14293::WARNING::2013-03-17 14:52:36,026::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:52:36,032::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:52:38,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:52:38,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881582 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:41,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:41,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798831 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:52:46,034::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:52:46,035::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:52:46,527::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:52:46,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:52:46,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:52:48,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:52:48,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012332 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:52:51,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:52:51,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099527 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:52:58,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:52:58,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166993 s, 245 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:53:01,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:53:01,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765396 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:53:08,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:53:08,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833139 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:53:12,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:53:12,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927341 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:53:18,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:53:18,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085662 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:53:22,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:53:22,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138863 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:53:28,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:53:28,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813007 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:53:32,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:53:32,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760092 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:53:38,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:53:38,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141514 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:53:42,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:53:42,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146805 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14294::WARNING::2013-03-17 14:53:46,563::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:53:46,567::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:53:48,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:53:48,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919898 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:53:52,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:53:52,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872861 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:53:56,569::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:53:56,570::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:53:57,168::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:53:57,174::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:53:57,175::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:53:58,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:53:58,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103207 s, 397 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:54:02,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:54:02,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864403 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:54:08,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:54:08,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008599 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:54:12,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:54:12,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767015 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:54:18,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:54:18,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017458 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:54:22,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:54:22,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016604 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:54:28,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:54:28,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096589 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:54:32,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:54:32,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148176 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:54:38,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:54:38,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167616 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:54:42,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:54:42,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143791 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:54:48,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:54:48,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778635 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:54:52,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:54:52,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001995 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14295::WARNING::2013-03-17 14:54:57,207::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:54:57,213::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:54:58,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:54:58,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172553 s, 237 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:55:02,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:55:02,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922255 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:55:07,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:55:07,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:55:07,632::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:55:07,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:55:07,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:08,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:09,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011799 s, 347 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:55:12,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:55:13,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00543348 s, 754 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:19,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:19,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782823 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:55:23,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:55:23,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146628 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:29,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:29,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961928 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:55:33,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:55:33,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00071169 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:39,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:39,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784777 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:55:43,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:55:43,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143041 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:49,107::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 14:55:49,108::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:55:49,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:49,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0132887 s, 154 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:49,164::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0132887 s, 154 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 14:55:49,164::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 14:55:49,165::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 14:55:49,166::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 14:55:49,167::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:49,168::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:49,168::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:49,169::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:49,169::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:49,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:49,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:49,561::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:55:49,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 14:55:49,565::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 14:55:49,566::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 14:55:49,566::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 14:55:49,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:49,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:49,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:55:49,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134982 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:55:53,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:55:53,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146168 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:55:59,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:56:00,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772234 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:56:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:56:03,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149376 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14296::WARNING::2013-03-17 14:56:07,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:56:07,672::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:56:10,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:56:10,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856935 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:56:13,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:56:13,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112932 s, 3.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:56:17,673::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 14:56:17,675::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:56:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:56:18,168::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:56:18,175::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:56:18,176::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:56:20,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:56:20,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121813 s, 336 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:56:23,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:56:23,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145556 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:56:30,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:56:30,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176081 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:56:33,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:56:33,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134725 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:56:40,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:56:40,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179463 s, 228 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:56:43,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:56:43,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150169 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:56:50,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:56:50,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171611 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:56:53,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:56:53,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126732 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:57:00,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:57:00,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892584 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:03,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:03,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800077 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:57:10,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:57:10,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850191 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:13,970::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 14:57:13,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:57:13,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:14,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131913 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:14,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131913 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 14:57:14,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 14:57:14,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 14:57:14,020::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 14:57:14,021::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:57:14,022::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:57:14,022::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:57:14,023::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:57:14,023::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 14:57:14,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:14,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:14,452::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 14:57:14,453::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 14:57:14,456::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 14:57:14,457::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 14:57:14,458::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 14:57:14,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:14,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:14,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:14,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880286 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14297::WARNING::2013-03-17 14:57:18,202::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:57:18,206::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:57:20,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:57:20,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171393 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:24,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:24,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871625 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:57:28,208::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:57:28,209::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:57:28,671::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:57:28,679::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:57:28,681::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:57:30,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:57:30,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129905 s, 315 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:34,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:34,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891086 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:57:40,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:57:40,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139241 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:45,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:45,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114171 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:57:50,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:57:50,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950103 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:57:55,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:57:55,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00655664 s, 625 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:58:00,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:58:00,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792709 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:58:05,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:58:05,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015611 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:58:10,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:58:10,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150124 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:58:15,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:58:15,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014469 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:58:20,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:58:20,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141836 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:58:25,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:58:25,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977436 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14298::WARNING::2013-03-17 14:58:28,711::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:58:28,715::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:58:30,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:58:30,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084551 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:58:35,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:58:35,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178444 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:58:38,717::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:58:38,718::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:58:39,180::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:58:39,187::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:58:39,188::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:58:40,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:58:40,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148292 s, 276 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:58:45,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:58:45,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192858 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:58:50,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:58:50,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00352713 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:58:55,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:58:55,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845143 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:59:00,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:59:00,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174179 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:59:05,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:59:05,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165012 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:59:10,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:59:10,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137356 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:59:15,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:59:15,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865416 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:59:20,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:59:20,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204446 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:59:25,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:59:25,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141662 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 14:59:30,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:59:30,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159978 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:59:35,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:59:35,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774555 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14299::WARNING::2013-03-17 14:59:39,221::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 14:59:39,225::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 14:59:40,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:59:41,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0312018 s, 131 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:59:46,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:59:46,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797847 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 14:59:49,230::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 14:59:49,231::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 14:59:49,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 14:59:49,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 14:59:49,665::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 14:59:51,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 14:59:51,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106773 s, 384 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 14:59:56,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 14:59:56,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895229 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:01,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:01,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771394 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:00:06,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:00:06,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956634 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:11,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:11,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149895 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:00:16,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:00:16,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102975 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:21,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:21,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167394 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:00:26,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:00:26,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178849 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:31,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:31,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771973 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:00:36,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:00:36,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00529404 s, 774 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:41,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:41,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847869 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:00:46,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:00:46,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276239 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14300::WARNING::2013-03-17 15:00:49,692::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:00:49,700::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:00:51,294::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:00:51,295::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:00:51,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:51,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000803128 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:51,326::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000803128 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:00:51,327::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:00:51,327::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:00:51,330::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:00:51,331::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:00:51,332::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:00:51,332::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:00:51,333::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:00:51,333::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:00:51,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:51,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:51,769::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:00:51,770::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:00:51,773::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:00:51,774::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:00:51,774::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:00:51,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:52,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:00:52,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:00:52,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147452 s, 278 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:00:56,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:00:56,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902059 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:00:59,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:00:59,707::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:01:00,211::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:01:00,218::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:01:00,219::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:01:02,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:01:02,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166942 s, 245 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:01:06,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:01:06,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148694 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:01:12,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:01:12,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878209 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:01:16,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:01:16,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142293 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:01:22,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:01:22,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973534 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:01:26,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:01:26,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862842 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:01:32,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:01:32,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834841 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:01:36,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:01:36,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139572 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:01:42,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:01:42,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127394 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:01:46,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:01:46,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100111 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:01:52,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:01:52,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185152 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:01:56,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:01:57,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149757 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14301::WARNING::2013-03-17 15:02:00,246::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:02:00,250::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:02:02,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:02:02,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769027 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:07,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805167 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:02:10,252::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:02:10,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:02:10,254::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:02:10,723::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:02:10,730::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:02:10,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:02:12,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:02:12,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140066 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:17,156::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:02:17,157::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:02:17,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:17,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000827841 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:17,192::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000827841 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:02:17,192::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:02:17,193::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:02:17,194::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:02:17,195::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:02:17,196::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:02:17,196::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:02:17,197::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:02:17,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:02:17,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:17,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:17,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:02:17,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:02:17,633::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:02:17,635::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:02:17,635::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:02:17,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:18,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:18,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:18,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176702 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:02:22,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:02:22,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152943 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:28,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:28,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790092 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:02:32,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:02:32,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162929 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:38,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:38,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821103 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:02:42,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:02:42,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965097 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:48,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:48,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789591 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:02:52,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:02:52,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173533 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:02:58,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:02:58,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088614 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:03:02,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:03:02,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.029649 s, 138 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:03:08,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:03:08,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828596 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14302::WARNING::2013-03-17 15:03:10,757::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:03:10,761::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:03:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:03:12,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861752 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:03:18,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:03:18,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796425 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:03:20,763::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:03:20,765::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:03:21,204::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:03:21,213::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:03:21,214::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:03:22,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:03:22,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139642 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:03:28,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:03:28,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847438 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:03:32,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:03:32,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213696 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:03:38,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:03:38,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149548 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:03:42,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:03:42,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198166 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:03:48,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:03:48,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309343 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:03:53,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:03:53,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701995 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:03:58,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:03:58,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179658 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:04:03,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:04:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741428 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:04:08,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:04:08,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874726 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:04:13,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:04:13,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143852 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:04:19,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:04:19,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711134 s, 5.8 MB/s\n'; <rc> = 0 >Thread-14303::WARNING::2013-03-17 15:04:21,243::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:04:21,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:04:23,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:04:23,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855794 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:04:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:04:29,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781636 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:04:31,248::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:04:31,249::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:04:31,670::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:04:31,677::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:04:31,678::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:04:33,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:04:33,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198258 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:04:39,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:04:39,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788812 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:04:43,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:04:43,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151183 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:04:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:04:49,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140335 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:04:53,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:04:53,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014044 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:04:59,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:04:59,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070603 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:03,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:03,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016001 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:05:09,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:05:09,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147229 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:13,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:13,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132595 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:05:19,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:05:19,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797174 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:23,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:23,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172372 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:05:29,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:05:29,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187438 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14304::WARNING::2013-03-17 15:05:31,704::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:05:31,708::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:05:33,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:33,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826075 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:05:39,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:05:39,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125748 s, 3.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:05:41,710::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:05:41,711::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:05:42,239::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:05:42,246::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:05:42,247::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:43,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:43,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00635805 s, 644 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:05:49,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:05:49,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816639 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:53,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:05:53,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:05:53,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:53,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00119694 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:53,639::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00119694 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:05:53,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:05:53,640::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:05:53,642::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:05:53,643::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:53,643::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:53,644::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:53,644::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:53,644::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:53,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:54,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:54,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:05:54,111::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:05:54,116::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:05:54,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:05:54,118::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:05:54,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:05:54,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:05:54,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00973496 s, 421 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:05:59,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:05:59,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155368 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:06:04,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:06:04,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257733 s, 159 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:06:09,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:06:09,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013818 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:06:14,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:06:14,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00261542 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:06:20,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:06:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805841 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:06:24,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:06:24,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797598 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:06:30,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:06:30,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00544052 s, 753 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:06:34,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:06:34,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174977 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:06:40,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:06:40,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710309 s, 5.8 MB/s\n'; <rc> = 0 >Thread-14305::WARNING::2013-03-17 15:06:42,291::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:06:42,296::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:06:44,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:06:44,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158858 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:06:50,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:06:50,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891309 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:06:52,298::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:06:52,299::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:06:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:06:52,763::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:06:52,765::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:06:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:06:54,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115508 s, 355 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:00,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:00,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726531 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:07:04,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:07:04,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141304 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:10,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:10,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864075 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:07:14,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:07:14,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078431 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:20,474::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:07:20,476::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:07:20,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:20,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00160585 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:20,512::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00160585 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:07:20,513::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:07:20,513::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:07:20,515::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:07:20,516::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:07:20,516::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:07:20,517::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:07:20,517::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:07:20,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:07:20,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:20,958::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:07:20,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:07:20,962::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:07:20,964::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:07:20,964::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:07:20,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:21,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:21,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:21,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147705 s, 277 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:07:24,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:07:25,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014406 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:31,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:31,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145653 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:07:35,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:07:35,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149436 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:41,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:41,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171793 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:07:45,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:07:45,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156128 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:07:51,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:07:51,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973759 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14306::WARNING::2013-03-17 15:07:52,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:07:52,798::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:07:55,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:07:55,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860378 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:08:01,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:08:02,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130909 s, 3.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:08:02,800::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:08:02,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:08:02,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:08:03,500::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:08:03,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:08:03,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:08:05,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:08:05,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117577 s, 348 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:08:12,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:08:12,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829956 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:08:15,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:08:15,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148333 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:08:22,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:08:22,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864913 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:08:25,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:08:25,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878862 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:08:32,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:08:32,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746885 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:08:35,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:08:35,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083327 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:08:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:08:42,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831107 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:08:45,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:08:45,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0315677 s, 130 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:08:52,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:08:52,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746158 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:08:55,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:08:55,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213598 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:09:02,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:09:02,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00633545 s, 647 kB/s\n'; <rc> = 0 >Thread-14307::WARNING::2013-03-17 15:09:03,536::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:09:03,540::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:09:05,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:09:05,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015397 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:09:12,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:09:12,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986947 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:09:13,542::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:09:13,543::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:09:13,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:09:13,985::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:09:13,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:09:15,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:09:15,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135158 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:09:22,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:09:22,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016588 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:09:25,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:09:25,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152986 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:09:32,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:09:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839294 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:09:35,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:09:35,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143651 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:09:42,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:09:42,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153329 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:09:45,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:09:45,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802857 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:09:52,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:09:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00994665 s, 412 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:09:55,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:09:55,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164802 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:10:03,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:10:03,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.024968 s, 164 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:05,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:05,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136114 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:10:13,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:10:13,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152012 s, 269 kB/s\n'; <rc> = 0 >Thread-14308::WARNING::2013-03-17 15:10:14,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:10:14,030::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:10:15,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:15,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082525 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:10:23,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:10:23,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00427709 s, 958 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:10:24,035::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:10:24,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:10:24,541::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:10:24,548::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:10:24,549::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:25,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014556 s, 281 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:10:33,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:10:33,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152288 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:35,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:35,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149239 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:10:43,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:10:43,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792496 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:45,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:46,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912888 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:10:54,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:10:54,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744422 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:56,008::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:10:56,010::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:10:56,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:56,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000854646 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:56,046::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000854646 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:10:56,046::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:10:56,046::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:10:56,049::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:10:56,050::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:56,051::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:56,051::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:56,051::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:56,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:56,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:56,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:56,495::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:10:56,496::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:10:56,499::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:10:56,500::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:10:56,500::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:10:56,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:56,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:10:56,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:10:56,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118916 s, 344 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:11:04,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:04,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155522 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:11:06,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:11:07,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809733 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:11:14,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:14,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017919 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:11:17,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:11:17,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990635 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:11:24,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:24,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012638 s, 324 kB/s\n'; <rc> = 0 >Thread-14309::WARNING::2013-03-17 15:11:24,583::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:11:24,595::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:11:27,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:11:27,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764209 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:11:34,598::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:11:34,600::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:34,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:34,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189207 s, 216 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:11:35,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:11:35,076::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:11:35,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:11:37,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:11:37,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191779 s, 214 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:11:44,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:44,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808049 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:11:47,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:11:47,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800468 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:11:54,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:11:54,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987283 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:11:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:11:57,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00437983 s, 935 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:04,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:04,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203138 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:12:07,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:12:07,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00766725 s, 534 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:14,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:15,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728638 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:12:17,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:12:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771733 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:25,074::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:12:25,076::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:12:25,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:25,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0007818 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:25,114::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0007818 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:12:25,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:12:25,115::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:12:25,117::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:12:25,118::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:25,119::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:25,120::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:25,120::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:25,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:25,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:25,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:25,545::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:25,546::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:12:25,549::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:12:25,550::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:12:25,551::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:12:25,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:25,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:25,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:25,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00774697 s, 529 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:12:27,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:12:27,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119702 s, 342 kB/s\n'; <rc> = 0 >Thread-14310::WARNING::2013-03-17 15:12:35,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:12:35,109::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:12:35,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:36,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880693 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:12:37,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:12:37,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078689 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:12:45,119::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:12:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:12:45,577::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:12:45,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:12:45,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:12:46,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:46,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121073 s, 338 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:12:47,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:12:47,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148928 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:12:56,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:12:56,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162007 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:12:57,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:12:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018826 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:13:06,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:13:06,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778429 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:13:07,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:13:07,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148939 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:13:16,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:13:16,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166064 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:13:17,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:13:17,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017528 s, 234 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:13:26,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:13:26,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754415 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:13:27,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:13:27,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152587 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:13:36,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:13:36,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136287 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:13:37,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:13:37,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146974 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14311::WARNING::2013-03-17 15:13:45,610::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:13:45,614::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:13:46,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:13:46,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770636 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:13:47,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:13:47,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159816 s, 256 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:13:55,616::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:13:55,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:13:55,619::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:13:56,186::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:13:56,193::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:13:56,194::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:13:56,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:13:56,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118463 s, 346 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:13:57,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:13:57,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120208 s, 341 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:14:06,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:14:06,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014547 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:14:07,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:14:07,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153609 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:14:16,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:14:16,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794153 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:14:17,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:14:17,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143688 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:14:26,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:14:26,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843145 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:14:27,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:14:27,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806138 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:14:36,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:14:36,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231384 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:14:37,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:14:37,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281796 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:14:47,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:14:47,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956638 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:14:47,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:14:48,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153764 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14312::WARNING::2013-03-17 15:14:56,219::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:14:56,223::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:14:57,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:14:57,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161694 s, 253 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:14:58,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:14:58,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762321 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:15:06,225::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:15:06,226::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:15:06,742::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:15:06,750::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:15:06,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:15:07,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:15:07,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00895463 s, 457 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:08,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:08,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00935673 s, 438 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:15:17,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:15:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016354 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:18,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:18,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100527 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:15:27,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:15:27,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172204 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:28,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:28,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152113 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:15:37,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:15:37,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856291 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:38,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:38,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074511 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:15:47,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:15:47,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258265 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:48,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:48,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800926 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:15:57,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:15:57,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209344 s, 196 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:58,301::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:15:58,302::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:15:58,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:58,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00161255 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:58,350::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00161255 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:15:58,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:15:58,351::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:15:58,352::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:15:58,353::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:15:58,354::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:15:58,354::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:15:58,355::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:15:58,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:15:58,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:58,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:58,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:15:58,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:15:58,779::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:15:58,780::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:15:58,781::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:15:58,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:59,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:15:59,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:15:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138329 s, 296 kB/s\n'; <rc> = 0 >Thread-14313::WARNING::2013-03-17 15:16:06,777::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:16:06,783::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:16:07,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:16:07,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862179 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:16:09,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:16:09,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169262 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:16:16,785::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:16:16,786::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:16:17,250::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:16:17,257::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:16:17,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:16:17,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:16:17,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785193 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:16:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:16:19,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193309 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:16:27,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:16:27,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728293 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:16:29,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:16:29,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783106 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:16:37,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:16:37,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105599 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:16:39,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:16:39,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104698 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:16:47,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:16:47,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837505 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:16:49,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:16:49,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944151 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:16:58,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:16:58,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130235 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:16:59,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:16:59,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909192 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:08,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:08,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998233 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:17:09,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:17:09,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168906 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14314::WARNING::2013-03-17 15:17:17,287::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:17:17,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:17:18,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731709 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:17:19,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:17:19,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868992 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:17:27,298::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:17:27,299::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:17:27,834::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:17:27,841::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:17:27,842::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,292::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:17:28,295::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:17:28,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:28,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0230223 s, 89.0 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:28,354::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0230223 s, 89.0 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:17:28,355::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:17:28,355::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:17:28,357::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:17:28,358::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,358::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,359::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,359::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,360::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:28,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:28,717::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:17:28,718::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:17:28,720::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:17:28,721::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:17:28,722::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:17:28,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:29,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:29,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:29,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00566269 s, 723 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:17:29,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:17:29,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00343785 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:39,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:39,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843397 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:17:39,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:17:39,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982745 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:49,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:49,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821624 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:17:49,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:17:49,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102378 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:17:59,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:17:59,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800373 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:17:59,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:17:59,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806343 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:18:09,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:18:09,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854009 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:18:09,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:18:09,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987365 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:18:19,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:18:19,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150693 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:18:19,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:18:19,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153743 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14315::WARNING::2013-03-17 15:18:27,878::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:18:27,884::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:18:29,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:18:29,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083124 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:18:29,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:18:29,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831021 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:18:37,887::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:18:37,889::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:18:38,372::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:18:38,408::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:18:38,409::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:18:39,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:18:39,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143612 s, 285 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:18:39,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:18:39,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118067 s, 347 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:18:49,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:18:49,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691934 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:18:49,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:18:49,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123845 s, 331 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:18:59,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:18:59,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854766 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:18:59,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:18:59,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00728247 s, 562 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:19:09,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:19:09,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908299 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:19:09,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:19:10,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143387 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:19:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:19:19,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018464 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:19:20,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:19:20,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822609 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:19:30,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:19:30,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:19:30,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153066 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:19:30,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114385 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14316::WARNING::2013-03-17 15:19:38,434::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:19:38,437::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:19:40,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:19:40,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:19:40,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790968 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:19:40,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780384 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:19:48,445::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:19:48,446::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:19:48,447::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:19:48,893::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:19:48,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:19:48,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:19:50,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:19:50,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:19:50,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010909 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:19:50,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00952078 s, 430 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:20:00,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:20:00,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801236 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:20:00,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:20:00,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167766 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:20:10,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:20:10,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936135 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:20:10,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:20:10,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973165 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:20:20,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:20:20,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081838 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:20:20,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:20:20,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251105 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:20:30,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:20:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147165 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:20:30,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:20:30,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084254 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:20:40,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:20:40,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708099 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:20:40,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:20:40,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00248189 s, 1.7 MB/s\n'; <rc> = 0 >Thread-14317::WARNING::2013-03-17 15:20:48,929::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:20:48,934::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:20:50,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:20:50,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089875 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:20:50,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:20:50,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154612 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:20:58,937::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:20:58,938::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:20:59,372::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:20:59,379::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:20:59,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:00,572::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:21:00,575::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:21:00,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:00,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0272912 s, 75.0 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:00,634::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0272912 s, 75.0 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:21:00,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:21:00,635::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:21:00,637::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:21:00,638::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:00,638::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:00,639::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:00,639::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:00,640::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:00,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:00,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:00,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155522 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:01,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:01,082::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:21:01,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:21:01,085::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:21:01,086::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:21:01,087::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:21:01,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:01,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:01,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:01,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171061 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:21:10,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:11,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148928 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:11,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:11,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245895 s, 167 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:21:21,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:21,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790675 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:21,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:21,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128764 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:21:31,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:31,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168323 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:31,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:31,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819123 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:21:41,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:41,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136816 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:41,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:41,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157132 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:21:51,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:21:51,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178267 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:21:51,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:21:51,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140043 s, 292 kB/s\n'; <rc> = 0 >Thread-14318::WARNING::2013-03-17 15:21:59,410::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:21:59,414::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:22:01,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:01,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855216 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:22:01,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:01,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814259 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:22:09,416::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:22:09,417::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:22:09,868::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:22:09,875::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:22:09,876::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:11,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:11,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095626 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:22:11,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:11,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140403 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:21,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:21,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186615 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:22:21,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:21,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800873 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:31,591::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:22:31,592::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:22:31,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:31,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000774281 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:31,621::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000774281 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:22:31,621::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:22:31,622::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:22:31,624::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:22:31,625::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:31,626::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:31,626::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:31,626::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:31,627::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:31,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:31,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:32,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138462 s, 296 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:32,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:32,057::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:22:32,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:22:32,060::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:22:32,061::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:22:32,062::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:22:32,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:32,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:32,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:32,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00957405 s, 428 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:22:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:42,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158238 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:42,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:42,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210855 s, 194 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:22:52,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:22:52,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163129 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:22:52,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:22:52,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0421099 s, 97.3 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:23:02,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:23:02,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141008 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:23:02,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:23:02,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761176 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14319::WARNING::2013-03-17 15:23:09,904::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:23:09,911::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:23:12,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:23:12,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162627 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:23:12,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:23:12,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784767 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:23:19,915::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:23:19,916::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:23:20,344::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:23:20,351::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:23:20,353::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:23:22,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:23:22,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168578 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:23:22,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:23:22,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165877 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:23:32,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:23:32,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204817 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:23:32,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:23:33,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001663 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:23:42,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:23:42,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164855 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:23:43,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:23:43,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941248 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:23:52,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:23:52,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016621 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:23:53,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:23:53,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142992 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:24:02,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:24:02,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142437 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:24:03,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:24:03,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148078 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:24:12,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:24:12,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856389 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:24:13,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:24:13,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198617 s, 206 kB/s\n'; <rc> = 0 >Thread-14320::WARNING::2013-03-17 15:24:20,380::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:24:20,386::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:24:22,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:24:22,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825337 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:24:23,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:24:23,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00370432 s, 1.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:24:30,393::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:24:30,395::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:24:30,856::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:24:30,863::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:24:30,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:24:32,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:24:32,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019821 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:24:33,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:24:33,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018522 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:24:42,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:24:42,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013933 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:24:43,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:24:43,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215584 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:24:52,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:24:52,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285784 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:24:53,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:24:53,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730194 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:25:02,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:25:02,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133811 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:25:03,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:25:03,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148839 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:25:12,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:25:12,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146638 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:25:13,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:25:13,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156779 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:25:22,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:25:22,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836667 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:25:23,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:25:23,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951553 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14321::WARNING::2013-03-17 15:25:30,903::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:25:30,907::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:25:32,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:25:32,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139746 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:25:33,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:25:33,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703569 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:25:40,908::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:25:40,910::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:25:40,911::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:25:41,335::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:25:41,342::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:25:41,343::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:25:42,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:25:42,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166661 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:25:44,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:25:44,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124374 s, 329 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:25:52,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:25:53,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00245518 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:25:54,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:25:54,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847723 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:03,026::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:26:03,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:26:03,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:03,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000812395 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:03,055::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000812395 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:26:03,055::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:26:03,055::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:26:03,057::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:26:03,058::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:03,058::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:03,059::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:03,059::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:03,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:03,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:03,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:03,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:03,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:26:03,481::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:26:03,482::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:26:03,483::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:26:03,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:03,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:03,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:03,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00500484 s, 818 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:26:04,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:26:04,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180521 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:13,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00686261 s, 597 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:26:14,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:26:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017441 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:23,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:24,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0259625 s, 158 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:26:24,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:26:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144348 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:26:34,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:34,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180023 s, 228 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:26:34,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:26:34,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741676 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14322::WARNING::2013-03-17 15:26:41,370::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:26:41,374::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:26:44,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:44,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268847 s, 152 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:26:44,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:26:44,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802194 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:26:51,376::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:26:51,377::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:26:51,839::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:26:51,846::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:26:51,847::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:26:54,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:26:54,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105848 s, 387 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:26:54,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:26:54,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784083 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:27:04,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:27:04,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00688435 s, 595 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:04,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:04,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849719 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:27:14,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:27:14,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173047 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:14,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:14,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0298876 s, 137 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:27:24,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:27:24,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00764645 s, 536 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:24,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:24,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242867 s, 169 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:27:34,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:27:34,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170251 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:34,964::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:27:34,965::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:27:34,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:35,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133834 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:35,007::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133834 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:27:35,009::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:27:35,010::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:27:35,011::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:27:35,012::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:27:35,013::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:27:35,013::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:27:35,013::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:27:35,014::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:27:35,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:35,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:35,443::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:27:35,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:27:35,446::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:27:35,448::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:27:35,448::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:27:35,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:35,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:35,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:35,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135368 s, 303 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:27:44,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:27:44,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134772 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:45,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:46,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00806969 s, 508 kB/s\n'; <rc> = 0 >Thread-14323::WARNING::2013-03-17 15:27:51,891::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:27:51,894::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:27:54,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:27:54,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781859 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:27:56,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:27:56,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937377 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:28:01,895::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:28:01,897::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:28:02,500::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:28:02,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:28:02,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:28:04,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:28:04,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154515 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:28:06,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:28:06,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127359 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:28:14,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:28:14,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150694 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:28:16,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:28:16,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160317 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:28:24,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:28:24,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894334 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:28:26,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:28:26,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206641 s, 198 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:28:34,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:28:34,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140618 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:28:36,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:28:36,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00868452 s, 472 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:28:44,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:28:44,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139215 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:28:46,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:28:46,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219973 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:28:54,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:28:54,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865781 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:28:56,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:28:56,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172687 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14324::WARNING::2013-03-17 15:29:02,537::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:29:02,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:29:04,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:29:04,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156123 s, 262 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:29:06,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:29:06,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156173 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:29:12,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:29:12,545::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:29:12,962::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:29:12,969::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:29:12,970::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:29:14,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:29:14,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00438787 s, 933 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:29:16,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:29:16,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179079 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:29:24,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:29:24,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175917 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:29:26,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:29:26,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792907 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:29:34,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:29:34,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154612 s, 265 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:29:36,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:29:36,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190923 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:29:45,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:29:45,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087753 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:29:46,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:29:46,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149502 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:29:55,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:29:55,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145386 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:29:57,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:29:57,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808274 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:30:05,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:30:05,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145054 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:30:07,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:30:07,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116447 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14325::WARNING::2013-03-17 15:30:13,000::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:30:13,006::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:30:15,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:30:15,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882202 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:30:17,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:30:17,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828991 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:30:23,009::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:30:23,010::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:30:23,422::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:30:23,429::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:30:23,431::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:30:25,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:30:25,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115994 s, 353 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:30:27,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:30:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134829 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:30:35,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:30:35,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983539 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:30:37,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:30:37,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021427 s, 191 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:30:45,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:30:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001315 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:30:47,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:30:47,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223522 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:30:55,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:30:55,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171793 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:30:57,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:30:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096806 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:05,347::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:31:05,348::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:31:05,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:05,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014421 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:05,396::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014421 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:31:05,396::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:31:05,396::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:31:05,398::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:31:05,399::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:05,400::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:05,400::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:05,400::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:05,401::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:05,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:05,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:05,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:05,837::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:31:05,839::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:31:05,841::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:31:05,841::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:31:05,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:06,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:06,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:06,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187785 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:31:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:31:07,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236228 s, 173 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:16,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:16,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014397 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:31:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:31:17,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0256777 s, 160 kB/s\n'; <rc> = 0 >Thread-14326::WARNING::2013-03-17 15:31:23,468::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:31:23,471::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:31:26,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:26,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175143 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:31:27,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:31:27,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193405 s, 2.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:31:33,474::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:31:33,475::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:31:33,476::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:31:33,893::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:31:33,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:31:33,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:31:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:36,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159057 s, 258 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:31:37,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:31:37,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851366 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:46,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:46,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901902 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:31:47,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:31:48,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215108 s, 190 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:31:56,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:31:56,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846969 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:31:58,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:31:58,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897214 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:32:06,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:32:06,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146647 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:08,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:08,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154515 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:32:16,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:32:16,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013996 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:18,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:18,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846069 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:32:26,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:32:26,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016417 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:28,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:28,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156984 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14327::WARNING::2013-03-17 15:32:33,932::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:32:33,935::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:32:36,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:32:36,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837949 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:38,364::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:32:38,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:32:38,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:38,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152783 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:38,401::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152783 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:32:38,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:32:38,402::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:32:38,404::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:32:38,405::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:32:38,406::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:32:38,406::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:32:38,406::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:32:38,407::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:32:38,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:38,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:38,841::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:32:38,842::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:32:38,844::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:32:38,845::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:32:38,846::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:32:38,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:39,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:39,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00841416 s, 487 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:32:43,937::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:32:43,938::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:32:44,401::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:32:44,408::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:32:44,410::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:32:46,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:32:46,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107969 s, 379 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:49,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:49,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112876 s, 363 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:32:56,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:32:56,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125375 s, 327 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:32:59,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:32:59,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083484 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:33:06,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:33:06,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185472 s, 221 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:33:09,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:33:09,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175045 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:33:16,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:33:16,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144403 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:33:19,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:33:19,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742329 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:33:26,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:33:26,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146157 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:33:29,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:33:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828672 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:33:36,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:33:36,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110391 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:33:39,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:33:39,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229839 s, 178 kB/s\n'; <rc> = 0 >Thread-14328::WARNING::2013-03-17 15:33:44,437::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:33:44,441::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:33:46,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:33:47,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862231 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:33:49,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:33:49,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113253 s, 362 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:33:54,443::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:33:54,444::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:33:54,883::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:33:54,890::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:33:54,892::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:33:57,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:33:57,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070663 s, 580 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:33:59,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:33:59,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142719 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:34:07,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:34:07,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867662 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:34:09,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:34:09,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834528 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:34:17,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:34:17,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00338809 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:34:20,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:34:20,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154287 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:34:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:34:27,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157343 s, 260 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:34:30,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:34:30,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862389 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:34:37,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:34:37,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852605 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:34:40,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:34:40,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160763 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:34:47,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:34:47,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133972 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:34:50,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:34:50,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152652 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14329::WARNING::2013-03-17 15:34:54,925::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:34:54,927::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:34:57,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:34:57,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775867 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:35:00,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:35:00,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079873 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:35:04,929::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:35:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:35:05,348::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:35:05,355::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:35:05,356::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:35:07,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:35:07,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00964986 s, 424 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:35:10,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:35:10,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799792 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:35:17,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:35:17,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145387 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:35:20,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:35:20,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213683 s, 192 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:35:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:35:27,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00720461 s, 569 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:35:30,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:35:30,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713717 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:35:37,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:35:37,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897893 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:35:40,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:35:40,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846962 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:35:47,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:35:47,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143311 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:35:50,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:35:50,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203317 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:35:57,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:35:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077589 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:36:00,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:36:00,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124084 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14330::WARNING::2013-03-17 15:36:05,386::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:36:05,390::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:36:07,545::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:36:07,546::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:36:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:07,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000855886 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:07,576::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000855886 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:36:07,577::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:36:07,577::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:36:07,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:36:07,581::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:07,582::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:07,582::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:07,583::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:07,583::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:07,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:08,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:08,018::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:08,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:36:08,021::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:36:08,023::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:36:08,023::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:36:08,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:08,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:08,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:08,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160843 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:36:10,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:36:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822622 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:36:15,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:36:15,393::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:36:15,856::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:36:15,863::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:36:15,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:36:18,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:18,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00997645 s, 411 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:36:20,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:36:21,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0237864 s, 172 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:28,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:28,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015567 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:36:31,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:36:31,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143572 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:38,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:38,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100613 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:36:41,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:36:41,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133518 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:48,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165685 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:36:51,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:36:51,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805433 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:36:58,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:36:58,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799695 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:01,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:01,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142115 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:37:08,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:37:08,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801592 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:11,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:11,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136916 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14331::WARNING::2013-03-17 15:37:15,892::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:37:15,898::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:37:18,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:37:18,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132189 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:21,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:21,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891303 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:37:25,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:37:25,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:37:25,901::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:37:26,348::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:37:26,355::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:37:26,358::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:37:28,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:37:28,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114015 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:31,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:31,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839176 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:37:38,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:37:38,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141416 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:41,579::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:37:41,580::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:37:41,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:41,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0243492 s, 84.1 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:41,631::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0243492 s, 84.1 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:37:41,632::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:37:41,632::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:37:41,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:37:41,635::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:37:41,636::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:37:41,636::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:37:41,636::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:37:41,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:37:41,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:42,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:42,055::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:37:42,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:37:42,059::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:37:42,060::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:37:42,061::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:37:42,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:42,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:42,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:42,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911038 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:37:48,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:37:48,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839755 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:37:52,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:37:52,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944462 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:37:58,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:37:58,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098121 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:38:02,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:38:02,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146244 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:38:08,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:38:08,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165961 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:38:12,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:38:12,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837745 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:38:18,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:38:19,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136432 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:38:22,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:38:22,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152699 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14332::WARNING::2013-03-17 15:38:26,390::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:38:26,392::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:38:29,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:38:29,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771772 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:38:32,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:38:32,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101249 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:38:36,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:38:36,396::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:38:36,899::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:38:36,906::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:38:36,907::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:38:39,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:38:39,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179184 s, 229 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:38:42,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:38:42,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0262763 s, 156 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:38:49,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:38:49,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808022 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:38:52,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:38:53,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102438 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:38:59,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:38:59,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814846 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:39:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:39:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871112 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:39:09,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:39:09,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807044 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:39:13,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:39:13,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166069 s, 247 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:39:19,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:39:19,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013669 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:39:23,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:39:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761427 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:39:29,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:39:29,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014104 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:39:33,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:39:33,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218678 s, 187 kB/s\n'; <rc> = 0 >Thread-14333::WARNING::2013-03-17 15:39:36,937::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:39:36,946::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:39:39,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:39:39,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978263 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:39:43,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:39:43,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859631 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:39:46,949::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:39:46,950::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:39:47,391::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:39:47,398::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:39:47,399::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:39:49,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:39:49,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161586 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:39:53,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:39:53,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085475 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:39:59,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:39:59,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089168 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:40:03,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:40:03,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205815 s, 199 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:40:09,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:40:09,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725769 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:40:13,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:40:13,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851399 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:40:19,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:40:19,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843451 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:40:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:40:23,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177488 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:40:29,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:40:29,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152584 s, 268 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:40:33,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:40:33,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001605 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:40:39,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:40:39,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747402 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:40:43,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:40:43,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117923 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14334::WARNING::2013-03-17 15:40:47,425::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:40:47,428::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:40:49,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:40:49,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081682 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:40:53,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:40:53,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205992 s, 199 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:40:57,441::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:40:57,442::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:40:57,886::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:40:57,893::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:40:57,894::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:40:59,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:40:59,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101202 s, 405 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:41:04,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:41:04,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100458 s, 408 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:09,670::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:41:09,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:41:09,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:09,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00183996 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:09,716::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00183996 s, 1.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:41:09,716::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:41:09,717::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:41:09,718::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:41:09,719::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:41:09,720::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:41:09,720::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:41:09,720::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:41:09,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:41:09,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:10,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:10,175::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:41:10,175::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:41:10,178::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:41:10,179::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:41:10,179::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:41:10,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:10,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:10,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:10,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155017 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:41:14,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:41:14,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929314 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:20,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:20,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149806 s, 273 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:41:24,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:41:24,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130266 s, 314 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:30,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:30,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897933 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:41:34,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:41:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772167 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:40,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:40,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139667 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:41:44,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:41:44,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114476 s, 358 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:41:50,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:41:50,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741819 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:41:54,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:41:54,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152422 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14335::WARNING::2013-03-17 15:41:57,919::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:41:57,923::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:42:00,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:42:00,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278936 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:04,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:04,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920226 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:42:07,926::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:42:07,927::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:42:08,356::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:42:08,363::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:42:08,364::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:42:10,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:42:10,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00916066 s, 447 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:14,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:14,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861082 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:42:20,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:42:20,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076907 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:24,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:24,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00267998 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:42:30,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:42:30,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164114 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:34,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:34,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142203 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:42:40,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:42:40,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774012 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:44,789::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:42:44,790::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:42:44,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:44,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000759147 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:44,820::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000759147 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:42:44,820::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:42:44,821::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:42:44,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:42:44,823::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:42:44,824::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:42:44,825::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:42:44,825::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:42:44,825::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:42:44,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:45,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:45,251::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:42:45,252::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:42:45,255::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:42:45,256::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:42:45,256::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:42:45,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:45,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:45,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:45,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00666113 s, 615 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:42:50,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:42:51,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150233 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:42:55,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:42:55,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211739 s, 193 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:43:01,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:43:01,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372119 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:43:05,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:43:05,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887379 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14336::WARNING::2013-03-17 15:43:08,391::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:43:08,398::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:43:11,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:43:11,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811393 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:43:15,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:43:15,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783003 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:43:18,400::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:43:18,401::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:43:18,402::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:43:18,926::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:43:18,932::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:43:18,934::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:43:21,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:43:21,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00871553 s, 470 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:43:26,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:43:26,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110578 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:43:31,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:43:31,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159484 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:43:36,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:43:36,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877622 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:43:41,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:43:41,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079071 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:43:46,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:43:46,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170359 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:43:51,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:43:51,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142679 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:43:56,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:43:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147921 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:44:01,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:44:01,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109052 s, 376 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:44:06,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:44:06,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137925 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:44:11,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:44:11,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148393 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:44:16,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:44:16,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842986 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14337::WARNING::2013-03-17 15:44:18,961::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:44:18,964::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:44:21,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:44:21,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769879 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:44:26,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:44:26,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825464 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:44:28,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:44:28,971::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:44:29,389::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:44:29,396::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:44:29,398::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:44:31,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:44:31,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107683 s, 380 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:44:36,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:44:36,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736025 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:44:41,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:44:41,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814129 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:44:46,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:44:46,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138017 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:44:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:44:51,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873548 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:44:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:44:56,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782756 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:45:01,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:45:01,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152396 s, 269 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:45:06,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:45:06,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820004 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:45:11,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:45:11,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163313 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:45:16,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:45:16,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769857 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:45:21,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:45:21,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151106 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:45:26,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:45:26,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825883 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14338::WARNING::2013-03-17 15:45:29,431::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:45:29,438::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:45:31,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:45:31,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842761 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:45:37,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:45:37,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904425 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:45:39,441::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:45:39,443::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:45:39,918::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:45:39,925::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:45:39,926::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:45:41,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:45:41,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00899894 s, 455 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:45:47,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:45:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849788 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:45:51,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:45:51,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719788 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:45:57,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:45:57,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199404 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:01,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:01,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125623 s, 326 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:46:07,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:46:07,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756526 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:11,937::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:46:11,938::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:46:11,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:11,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000821466 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:11,991::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000821466 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:46:11,993::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:46:11,995::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:46:12,001::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:46:12,010::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:12,011::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:12,011::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:12,012::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:12,012::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:12,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:12,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:12,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:12,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:46:12,445::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:46:12,446::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:46:12,447::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:46:12,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:12,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:12,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:12,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144453 s, 284 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:46:17,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:46:17,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178104 s, 230 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:22,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:22,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126179 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:46:27,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:46:27,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236241 s, 173 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:46:32,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:32,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740353 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:46:37,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:46:37,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111859 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14339::WARNING::2013-03-17 15:46:39,952::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:46:39,957::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:46:42,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:43,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173314 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:46:47,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:46:47,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756618 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:46:49,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:46:49,962::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:46:50,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:46:50,425::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:46:50,426::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:46:53,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:46:53,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0081718 s, 501 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:46:57,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:46:57,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877562 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:47:03,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:47:03,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218156 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:07,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:07,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201715 s, 203 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:47:13,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:47:13,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00780881 s, 525 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:17,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:17,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202042 s, 203 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:47:23,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:47:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156184 s, 262 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:27,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:27,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748255 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:47:33,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:47:33,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771927 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:38,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:38,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159584 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:47:43,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:47:43,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161206 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:48,103::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:47:48,104::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:47:48,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:48,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129239 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:48,160::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129239 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:47:48,162::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:47:48,162::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:47:48,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:47:48,165::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:47:48,167::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:47:48,168::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:47:48,168::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:47:48,169::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:47:48,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:48,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:48,606::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:47:48,607::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:47:48,610::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:47:48,611::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:47:48,611::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:47:48,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:49,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:49,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:49,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142428 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14340::WARNING::2013-03-17 15:47:50,454::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:47:50,462::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:47:53,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:47:53,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780108 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:47:59,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:47:59,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780634 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:48:00,465::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:48:00,466::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:48:01,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:48:01,019::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:48:01,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:48:03,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:48:03,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00898403 s, 456 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:48:09,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:48:09,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00386557 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:48:13,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:48:13,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894615 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:48:19,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:48:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991977 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:48:23,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:48:23,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713259 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:48:29,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:48:29,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214909 s, 191 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:48:33,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:48:33,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868827 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:48:39,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:48:39,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00349788 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:48:43,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:48:43,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161429 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:48:49,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:48:49,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808173 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:48:53,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:48:53,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141185 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:48:59,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:48:59,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082157 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14341::WARNING::2013-03-17 15:49:01,048::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:49:01,052::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:49:03,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:49:03,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080023 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:49:09,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:49:09,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106234 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:49:11,054::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:49:11,055::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:49:11,056::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:49:11,551::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:49:11,558::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:49:11,560::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:49:13,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:49:13,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127886 s, 320 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:49:19,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:49:19,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159442 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:49:23,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:49:23,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833009 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:49:29,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:49:29,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173442 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:49:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:49:33,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014021 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:49:39,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:49:39,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842042 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:49:43,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:49:43,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168209 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:49:49,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:49:49,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940214 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:49:53,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:49:53,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134967 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:50:00,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:50:00,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016337 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:50:03,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:50:03,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014872 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:50:10,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:50:10,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107875 s, 3.8 MB/s\n'; <rc> = 0 >Thread-14342::WARNING::2013-03-17 15:50:11,589::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:50:11,594::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:50:13,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:50:13,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758479 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:50:20,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:50:20,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835677 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:50:21,597::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:50:21,600::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:50:22,106::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:50:22,113::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:50:22,114::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:50:23,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:50:23,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00723274 s, 566 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:50:30,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:50:30,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824697 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:50:33,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:50:34,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147326 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:50:40,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:50:40,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008891 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:50:44,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:50:44,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149116 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:50:50,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:50:50,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014483 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:50:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:50:54,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114605 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:51:00,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:51:00,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813821 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:04,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:04,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133385 s, 307 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:51:10,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:51:10,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131003 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:14,188::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:51:14,193::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:51:14,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:14,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000961215 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:14,226::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000961215 s, 2.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:51:14,227::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:51:14,227::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:51:14,228::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:51:14,229::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:14,230::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:14,230::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:14,231::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:14,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:14,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:14,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:14,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:14,663::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:51:14,665::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:51:14,667::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:51:14,667::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:51:14,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:15,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:15,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105487 s, 388 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:51:20,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:51:20,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109411 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14343::WARNING::2013-03-17 15:51:22,142::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:51:22,148::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:51:25,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:25,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817779 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:51:30,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:51:30,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220702 s, 186 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:51:32,150::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:51:32,151::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:51:32,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:51:32,683::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:51:32,684::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:51:35,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:35,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137854 s, 297 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:51:40,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:51:40,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898554 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:45,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761592 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:51:50,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:51:50,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232054 s, 177 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:51:55,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:51:55,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949085 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:01,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:01,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809913 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:52:05,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:52:05,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085607 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:11,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:11,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755967 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:52:15,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:52:15,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902544 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:21,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:21,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103896 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:52:25,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:52:25,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082551 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:31,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:31,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100241 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14344::WARNING::2013-03-17 15:52:32,711::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:52:32,718::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:52:35,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:52:35,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842363 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:41,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:41,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867386 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:52:42,726::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:52:42,727::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:52:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:52:43,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:52:43,236::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:52:45,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:52:45,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116523 s, 352 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:51,394::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:52:51,395::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:52:51,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:51,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000716174 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:51,426::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000716174 s, 2.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:52:51,426::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:52:51,426::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:52:51,428::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:52:51,429::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:52:51,430::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:52:51,430::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:52:51,431::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:52:51,431::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:52:51,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:51,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:51,880::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:52:51,881::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:52:51,884::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:52:51,885::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:52:51,886::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:52:51,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:52,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:52:52,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:52:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010814 s, 379 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:52:55,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:52:55,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146002 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:53:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:53:02,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806112 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:53:05,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:53:05,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148779 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:53:12,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:53:12,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143573 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:53:15,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:53:15,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074204 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:53:22,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:53:22,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169602 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:53:25,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:53:25,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795494 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:53:32,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:53:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156063 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:53:35,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:53:35,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151547 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:53:42,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:53:42,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963474 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14345::WARNING::2013-03-17 15:53:43,282::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:53:43,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:53:45,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:53:45,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008162 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:53:52,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:53:53,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00903833 s, 453 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:53:53,288::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:53:53,290::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:53:53,754::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:53:53,760::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:53:53,762::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:53:55,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:53:55,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147406 s, 278 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:54:03,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:54:03,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811541 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:54:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:54:05,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824192 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:54:13,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:54:13,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136825 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:54:15,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:54:15,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756994 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:54:23,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:54:23,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763487 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:54:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:54:25,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256271 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:54:33,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:54:33,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196682 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:54:35,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:54:35,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148526 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:54:43,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:54:43,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088991 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:54:45,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:54:46,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170761 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:54:53,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:54:53,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816448 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14346::WARNING::2013-03-17 15:54:53,792::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:54:53,799::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 15:54:56,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:54:56,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157606 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:55:03,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 15:55:03,808::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 15:55:03,829::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:55:03,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905664 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:55:03,844::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:55:04,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:55:04,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:55:04,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:55:06,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:55:06,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108352 s, 378 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:55:14,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:55:14,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177823 s, 230 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:55:16,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:55:16,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129865 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:55:24,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:55:24,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108547 s, 377 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:55:26,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:55:26,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095012 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:55:34,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:55:34,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142271 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:55:36,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:55:36,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149916 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:55:44,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:55:44,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0295124 s, 139 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:55:46,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:55:46,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178231 s, 230 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:55:54,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:55:54,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168204 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:55:56,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:55:56,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887483 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14347::WARNING::2013-03-17 15:56:04,544::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:56:04,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:56:04,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:04,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245026 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:06,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:06,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168585 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:56:14,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:56:14,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:14,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:14,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150891 s, 271 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:56:15,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:56:15,072::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:56:15,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,402::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 15:56:16,405::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:56:16,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:16,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.012551 s, 163 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:16,448::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.012551 s, 163 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 15:56:16,449::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 15:56:16,449::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 15:56:16,451::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 15:56:16,453::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,454::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,454::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,455::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,455::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:16,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:16,874::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:56:16,875::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 15:56:16,879::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 15:56:16,880::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 15:56:16,881::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 15:56:16,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:17,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:17,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:17,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00634369 s, 646 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:56:24,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:24,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735508 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:27,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:27,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117255 s, 349 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:56:34,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:34,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145986 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:37,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:37,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122787 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:56:44,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:44,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152034 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:47,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:47,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799357 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:56:55,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:56:55,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877815 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:56:57,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:56:57,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142082 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:05,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:05,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810071 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:57:07,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:57:07,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151744 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14348::WARNING::2013-03-17 15:57:15,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:57:15,104::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:57:15,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:15,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158179 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:57:17,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:57:17,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00390112 s, 1.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:57:25,106::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:57:25,107::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:25,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:25,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156411 s, 262 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:57:25,596::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:57:25,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:57:25,603::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 15:57:27,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:57:27,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193758 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:35,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:35,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976194 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:57:37,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:57:37,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158157 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:45,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:45,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885271 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:57:47,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:57:47,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143218 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:55,503::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 15:57:55,504::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:57:55,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:55,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0213646 s, 95.9 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:55,560::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0213646 s, 95.9 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 15:57:55,560::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 15:57:55,560::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 15:57:55,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 15:57:55,564::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:57:55,565::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:57:55,565::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:57:55,565::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:57:55,566::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 15:57:55,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:55,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:55,969::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:57:55,970::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 15:57:55,972::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 15:57:55,974::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 15:57:55,974::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 15:57:55,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:56,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:57:56,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:57:56,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109294 s, 375 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:57:57,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:57:57,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00472033 s, 868 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:58:06,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:58:06,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143609 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:58:07,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:58:07,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791689 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:58:16,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:58:16,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134729 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:58:17,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:58:17,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165698 s, 247 kB/s\n'; <rc> = 0 >Thread-14349::WARNING::2013-03-17 15:58:25,628::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:58:25,634::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:58:26,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:58:26,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008881 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:58:27,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:58:27,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804135 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:58:35,636::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:58:35,637::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:58:36,056::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:58:36,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:58:36,065::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:58:36,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:58:36,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126336 s, 324 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:58:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:58:37,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100288 s, 408 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:58:46,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:58:46,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0237226 s, 173 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:58:47,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:58:47,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902921 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:58:56,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:58:56,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090363 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:58:57,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:58:57,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242466 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:59:06,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:59:07,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144532 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:59:07,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:59:08,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00262259 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:59:17,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:59:17,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160683 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:59:18,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:59:18,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289022 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:59:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:59:27,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165538 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:59:28,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:59:28,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137669 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14350::WARNING::2013-03-17 15:59:36,092::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 15:59:36,097::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 15:59:37,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:59:37,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219817 s, 186 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:59:38,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:59:38,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173336 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 15:59:46,103::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 15:59:46,104::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 15:59:46,589::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 15:59:46,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 15:59:46,597::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 15:59:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:59:47,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00707999 s, 579 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:59:48,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:59:48,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131741 s, 311 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 15:59:57,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 15:59:57,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700241 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 15:59:58,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 15:59:58,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164118 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:00:07,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:00:07,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00533322 s, 768 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:00:08,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:00:08,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886051 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:00:17,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:00:17,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00943818 s, 434 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:00:18,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:00:18,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165848 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:00:27,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:00:27,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156405 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:00:28,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:00:28,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855329 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:00:37,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:00:37,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226827 s, 181 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:00:38,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:00:38,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780146 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14351::WARNING::2013-03-17 16:00:46,630::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:00:46,634::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:00:47,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:00:47,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847965 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:00:48,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:00:48,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200006 s, 205 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:00:56,636::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:00:56,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:00:56,638::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:00:57,063::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:00:57,070::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:00:57,071::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:00:57,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:00:57,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206182 s, 199 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:00:58,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:00:58,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968302 s, 423 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:01:07,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:01:08,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695408 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:08,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:08,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829881 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:01:18,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:01:18,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155898 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:18,583::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:01:18,587::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:01:18,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:18,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0154727 s, 132 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:18,635::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0154727 s, 132 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:01:18,636::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:01:18,636::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:01:18,638::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:01:18,639::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:01:18,639::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:01:18,640::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:01:18,640::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:01:18,640::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:01:18,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:19,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:19,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:01:19,126::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:01:19,128::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:01:19,130::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:01:19,130::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:01:19,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:19,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:19,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:19,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00646873 s, 633 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:01:28,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:01:28,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148444 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:29,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:29,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168225 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:01:38,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:01:38,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169095 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:39,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:39,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015795 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:01:48,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:01:48,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128717 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:49,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225913 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14352::WARNING::2013-03-17 16:01:57,104::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:01:57,108::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:01:58,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:01:58,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000642066 s, 6.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:01:59,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:01:59,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945388 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:02:07,123::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:02:07,124::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:02:07,563::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:02:07,569::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:02:07,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:08,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:08,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00321749 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:02:09,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:02:09,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724066 s, 566 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:18,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:18,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181115 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:02:19,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:02:19,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832431 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:28,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:28,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014272 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:02:29,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:02:29,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876648 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:38,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:38,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807494 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:02:39,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:02:39,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737304 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:48,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:48,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146443 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:02:49,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:02:49,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149987 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:58,852::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:02:58,854::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:02:58,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:58,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137061 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:58,887::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137061 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:02:58,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:02:58,888::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:02:58,890::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:02:58,891::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:58,894::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:58,894::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:58,894::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:58,895::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:58,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:59,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:59,317::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:02:59,318::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:02:59,321::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:02:59,323::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:02:59,323::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:02:59,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:59,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:02:59,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:02:59,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00989818 s, 414 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:02:59,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:02:59,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207961 s, 197 kB/s\n'; <rc> = 0 >Thread-14353::WARNING::2013-03-17 16:03:07,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:03:07,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:03:09,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:03:09,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792284 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:03:10,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:03:10,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879825 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:03:17,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:03:17,618::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:03:18,059::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:03:18,066::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:03:18,068::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:03:19,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:03:19,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0069885 s, 586 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:03:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:03:20,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150206 s, 273 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:03:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:03:29,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787459 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:03:30,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:03:30,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880729 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:03:39,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:03:40,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795165 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:03:40,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:03:40,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838651 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:03:50,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:03:50,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136778 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:03:50,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:03:50,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810173 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:04:00,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:04:00,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151487 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:04:00,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:04:00,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103088 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:04:10,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:04:10,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147265 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:04:10,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:04:10,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991277 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14354::WARNING::2013-03-17 16:04:18,112::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:04:18,115::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:04:20,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:04:20,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:04:20,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120565 s, 340 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:04:20,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151406 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:04:28,117::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:04:28,118::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:04:28,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:04:28,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:04:28,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:04:30,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:04:30,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:04:30,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172135 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:04:30,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127235 s, 322 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:04:40,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:04:40,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757617 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:04:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:04:40,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196979 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:04:50,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:04:50,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936172 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:04:50,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:04:50,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001529 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:05:00,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:05:00,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144666 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:05:00,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:05:00,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0590743 s, 69.3 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:05:10,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:05:10,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719747 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:05:10,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:05:10,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250689 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:05:20,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:05:20,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213526 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:05:20,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:05:20,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159056 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14355::WARNING::2013-03-17 16:05:28,670::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:05:28,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:05:30,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:05:30,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756254 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:05:31,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:05:31,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870222 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:05:38,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:05:38,679::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:05:39,121::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:05:39,128::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:05:39,129::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:05:40,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:05:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158233 s, 259 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:05:41,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:05:41,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115863 s, 354 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:05:50,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:05:50,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107343 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:05:51,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:05:51,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864339 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:00,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:00,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150622 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:06:01,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:06:01,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932353 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:10,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851371 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:06:11,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:06:11,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903405 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:20,922::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:06:20,923::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:06:20,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:20,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762905 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:20,956::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762905 s, 2.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:06:20,956::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:06:20,956::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:06:20,958::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:06:20,959::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:06:20,960::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:06:20,960::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:06:20,960::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:06:20,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:06:20,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:21,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:21,397::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:06:21,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:21,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:06:21,450::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:06:21,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:06:21,459::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:06:21,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191974 s, 213 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:21,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:21,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:21,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:21,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108493 s, 378 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:06:31,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:06:31,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811836 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:31,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:31,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142877 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14356::WARNING::2013-03-17 16:06:39,156::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:06:39,160::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:06:41,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:06:41,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855268 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:42,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:42,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798234 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:06:49,161::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:06:49,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:06:49,164::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:06:49,631::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:06:49,638::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:06:49,639::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:06:51,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:06:51,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817149 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:06:52,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:06:52,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118305 s, 346 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:07:01,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:07:01,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820798 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:07:02,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:07:02,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278535 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:07:11,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:07:11,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158994 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:07:12,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:07:12,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213182 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:07:21,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:07:21,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135601 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:07:22,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:07:22,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879896 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:07:32,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:07:32,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984435 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:07:32,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:07:32,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822543 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:07:42,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:07:42,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018628 s, 220 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:07:42,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:07:42,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285447 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14357::WARNING::2013-03-17 16:07:49,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:07:49,676::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:07:52,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:07:52,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862601 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:07:52,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:07:52,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085568 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:07:59,679::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:07:59,680::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:08:00,132::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:08:00,138::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:08:00,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,224::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:08:02,228::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:08:02,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:02,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000806441 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:02,256::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000806441 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:08:02,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:08:02,257::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:08:02,259::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:08:02,261::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,262::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,262::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,263::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,263::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:02,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:02,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164523 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:02,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:08:02,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:08:02,701::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:08:02,702::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:08:02,702::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:08:02,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:03,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:03,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:03,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00691696 s, 592 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:08:12,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:12,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144605 s, 283 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:13,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:13,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145809 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:08:22,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:22,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802624 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:23,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007983 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:08:32,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:32,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746653 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:33,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:33,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927517 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:08:42,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:42,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750396 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:43,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:43,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148266 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:08:52,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:08:52,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802802 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:08:53,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:08:53,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203613 s, 201 kB/s\n'; <rc> = 0 >Thread-14358::WARNING::2013-03-17 16:09:00,166::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:09:00,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:09:02,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:09:02,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084074 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:09:03,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:09:03,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923792 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:09:10,172::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:09:10,173::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:09:10,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:09:10,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:09:10,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:09:12,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:09:12,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195532 s, 209 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:09:13,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:09:13,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894614 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:09:22,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:09:22,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00740336 s, 553 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:09:23,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:09:23,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210752 s, 194 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:09:32,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:09:32,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129807 s, 316 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:09:33,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:09:33,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142908 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:09:42,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:09:42,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149558 s, 274 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:09:43,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:09:43,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809707 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:09:52,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:09:52,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135611 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:09:53,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:09:53,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139895 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:10:02,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:10:02,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759339 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:10:04,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:10:04,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019641 s, 209 kB/s\n'; <rc> = 0 >Thread-14359::WARNING::2013-03-17 16:10:10,674::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:10:10,679::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:10:12,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:10:12,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852164 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:10:14,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:10:14,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824848 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:10:20,682::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:10:20,683::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:10:21,137::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:10:21,144::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:10:21,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:10:23,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:10:23,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00365871 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:10:24,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:10:24,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131804 s, 311 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:10:33,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:10:33,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096197 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:10:34,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:10:34,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083473 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:10:43,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:10:43,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840713 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:10:44,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:10:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.032685 s, 125 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:10:53,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:10:53,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836815 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:10:54,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:10:54,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739037 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:03,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:03,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141848 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:11:04,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:11:04,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139404 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:13,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:13,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001584 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:11:14,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:11:14,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015764 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14360::WARNING::2013-03-17 16:11:21,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:11:21,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:11:23,245::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:11:23,246::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:11:23,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:23,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00104065 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:23,277::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00104065 s, 2.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:11:23,277::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:11:23,277::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:11:23,279::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:11:23,280::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:23,281::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:23,281::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:23,282::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:23,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:23,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:23,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:23,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:23,723::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:11:23,725::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:11:23,726::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:11:23,727::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:11:23,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:24,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:24,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:24,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175678 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:11:24,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:11:24,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122521 s, 334 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:11:31,189::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:11:31,190::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:11:31,654::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:11:31,660::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:11:31,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:11:34,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:34,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00643827 s, 636 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:11:34,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:11:34,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158798 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:44,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:44,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158457 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:11:44,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:11:44,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001673 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:11:54,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:11:54,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159798 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:11:55,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:11:55,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923901 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:12:04,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:12:04,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116779 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:12:05,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:12:05,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138886 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:12:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:12:14,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150026 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:12:15,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:12:15,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758296 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:12:24,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:12:24,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105561 s, 388 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:12:25,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:12:25,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146292 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14361::WARNING::2013-03-17 16:12:31,691::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:12:31,697::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:12:34,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:12:34,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841951 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:12:35,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:12:35,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101464 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:12:41,699::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:12:41,700::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:12:41,701::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:12:42,127::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:12:42,134::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:12:42,136::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:12:44,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:12:44,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121324 s, 338 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:12:45,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:12:45,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802841 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:12:54,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:12:54,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145007 s, 282 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:12:55,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:12:55,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855826 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:13:04,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:13:04,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151554 s, 270 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:05,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:13:05,549::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:13:05,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:05,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00085106 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:05,577::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00085106 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:13:05,577::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:13:05,578::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:13:05,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:13:05,580::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:13:05,581::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:13:05,582::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:13:05,582::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:13:05,582::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:13:05,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:06,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:06,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:13:06,065::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:13:06,068::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:13:06,069::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:13:06,069::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:13:06,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:06,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:06,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:06,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00709938 s, 577 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:13:14,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:13:14,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185842 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:16,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:16,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185962 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:13:24,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:13:24,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863432 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:26,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814492 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:13:34,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:13:34,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169465 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:36,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:36,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136824 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14362::WARNING::2013-03-17 16:13:42,162::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:13:42,166::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:13:44,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:13:44,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873687 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:46,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:46,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226953 s, 180 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:13:52,167::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:13:52,169::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:13:52,647::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:13:52,654::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:13:52,655::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:13:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:13:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0084312 s, 486 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:13:56,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:13:56,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155544 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:14:04,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:14:04,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824803 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:14:06,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:14:06,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00522497 s, 784 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:14:14,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:14:14,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948518 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:14:17,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:14:17,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989277 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:14:24,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:14:24,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159113 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:14:27,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:14:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197306 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:14:34,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:14:35,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289927 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:14:37,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:14:37,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013453 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:14:45,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:14:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270569 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:14:47,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:14:47,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140861 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14363::WARNING::2013-03-17 16:14:52,686::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:14:52,695::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:14:55,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:14:55,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104917 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:14:57,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:14:57,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016293 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:15:02,698::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:15:02,699::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:15:03,155::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:15:03,161::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:15:03,162::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:15:05,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:15:05,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017588 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:15:07,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:15:07,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872775 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:15:15,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:15:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814901 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:15:17,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:15:17,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232366 s, 176 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:15:25,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:15:25,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825913 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:15:27,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:15:27,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224642 s, 182 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:15:35,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:15:35,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876084 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:15:37,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:15:37,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862074 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:15:45,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:15:45,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145665 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:15:47,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:15:47,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013908 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:15:55,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:15:55,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203568 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:15:57,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:15:57,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741295 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14364::WARNING::2013-03-17 16:16:03,193::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:16:03,197::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:16:05,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:05,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781413 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:16:07,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:16:07,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806253 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:16:13,199::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:16:13,201::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:16:13,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:16:13,683::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:16:13,685::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:15,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:15,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102598 s, 399 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:16:17,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:16:18,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00237097 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:25,559::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:16:25,560::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:16:25,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:25,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000846118 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:25,593::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000846118 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:16:25,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:16:25,593::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:16:25,595::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:16:25,596::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:25,596::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:25,597::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:25,597::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:25,598::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:25,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:26,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:26,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:16:26,075::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:16:26,077::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:16:26,078::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:16:26,079::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:16:26,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:26,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:26,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:26,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00580096 s, 706 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:16:28,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:16:28,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158677 s, 258 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:36,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:36,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193089 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:16:38,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:16:38,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907482 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:46,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:46,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151455 s, 270 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:16:48,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:16:48,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865981 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:16:56,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:16:56,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116596 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:16:58,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:16:58,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137908 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:17:06,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:17:06,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015016 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:17:08,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:17:08,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131224 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14365::WARNING::2013-03-17 16:17:13,712::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:17:13,717::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:17:16,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:17:16,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812459 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:17:18,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:17:18,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216057 s, 190 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:17:23,720::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:17:23,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:17:24,163::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:17:24,172::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:17:24,174::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:17:26,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:17:26,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703775 s, 582 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:17:28,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:17:28,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146497 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:17:36,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:17:36,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806987 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:17:38,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:17:38,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309792 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:17:46,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:17:46,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150137 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:17:48,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:17:48,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129782 s, 316 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:17:56,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:17:56,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813881 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:17:58,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:17:58,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781999 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:18:06,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:18:06,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187159 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:08,879::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:18:08,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:18:08,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:08,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.001036 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:08,919::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.001036 s, 2.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:18:08,919::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:18:08,920::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:18:08,921::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:18:08,922::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:18:08,923::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:18:08,924::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:18:08,924::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:18:08,924::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:18:08,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:09,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:09,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:18:09,426::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:18:09,428::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:18:09,429::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:18:09,430::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:18:09,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:09,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:09,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014733 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:18:16,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:18:17,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149572 s, 274 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:19,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00689735 s, 594 kB/s\n'; <rc> = 0 >Thread-14366::WARNING::2013-03-17 16:18:24,204::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:18:24,209::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:18:27,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:18:27,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153841 s, 266 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:29,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:29,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795444 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:18:34,211::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:18:34,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:18:34,213::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:18:34,750::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:18:34,757::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:18:34,758::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:18:37,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:18:37,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177267 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:40,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:40,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833552 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:18:47,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:18:47,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092293 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:18:50,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:18:50,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141456 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:18:57,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:18:57,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158836 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:19:00,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:19:00,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168746 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:19:07,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:19:07,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785711 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:19:10,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:19:10,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889534 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:19:17,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:19:17,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016088 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:19:20,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:19:20,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148623 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:19:27,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:19:27,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189935 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:19:30,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:19:30,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152213 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14367::WARNING::2013-03-17 16:19:34,789::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:19:34,794::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:19:37,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:19:37,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835693 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:19:40,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:19:40,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223801 s, 183 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:19:44,797::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:19:44,798::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:19:45,225::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:19:45,232::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:19:45,234::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:19:47,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:19:47,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130498 s, 314 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:19:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:19:50,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164934 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:19:57,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:19:57,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158121 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:20:00,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:20:00,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00277078 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:20:07,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:20:07,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788332 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:20:10,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:20:10,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960861 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:20:17,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:20:17,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791711 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:20:20,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:20:20,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111352 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:20:27,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:20:27,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121996 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:20:30,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:20:30,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147241 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:20:37,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:20:37,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010785 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:20:40,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:20:41,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854101 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14368::WARNING::2013-03-17 16:20:45,273::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:20:45,279::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:20:47,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:20:47,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154903 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:20:51,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:20:51,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131843 s, 311 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:20:55,281::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:20:55,282::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:20:55,786::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:20:55,793::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:20:55,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:20:57,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:20:57,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00712228 s, 575 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:21:01,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:21:01,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854009 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:07,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:07,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993263 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:21:11,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:21:11,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158703 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:17,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:17,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147677 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:21:21,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:21:21,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00614839 s, 666 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:27,874::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:21:27,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:21:27,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:27,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013739 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:27,907::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013739 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:21:27,907::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:21:27,908::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:21:27,909::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:21:27,910::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:21:27,911::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:21:27,911::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:21:27,912::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:21:27,912::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:21:27,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:28,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:28,351::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:21:28,352::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:21:28,355::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:21:28,356::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:21:28,356::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:21:28,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:28,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:28,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:28,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168626 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:21:31,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:21:31,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769933 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:38,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:38,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819861 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:21:41,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:21:41,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816805 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:21:48,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:48,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131663 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:21:51,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:21:51,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001116 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14369::WARNING::2013-03-17 16:21:55,823::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:21:55,829::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:21:58,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:21:58,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256429 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:22:01,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:22:01,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224388 s, 183 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:22:05,839::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:22:05,840::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:22:06,256::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:22:06,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:22:06,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:22:08,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:22:08,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106191 s, 386 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:22:11,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:22:11,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140392 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:22:18,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:22:19,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148554 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:22:21,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:22:21,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145119 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:22:29,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:22:29,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136411 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:22:31,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:22:31,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758131 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:22:39,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:22:39,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146204 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:22:41,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:22:41,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191003 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:22:49,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:22:49,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001796 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:22:52,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:22:52,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306107 s, 134 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:22:59,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:22:59,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140532 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:02,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:02,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191824 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14370::WARNING::2013-03-17 16:23:06,293::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:23:06,297::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:23:09,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:23:09,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896761 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:12,224::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:23:12,225::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:23:12,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:12,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000795169 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:12,253::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000795169 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:23:12,253::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:23:12,254::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:23:12,255::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:23:12,256::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:23:12,257::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:23:12,258::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:23:12,258::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:23:12,258::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:23:12,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:12,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:12,689::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:23:12,690::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:23:12,692::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:23:12,693::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:23:12,694::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:23:12,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:13,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:13,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:13,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449432 s, 911 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:23:16,299::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:23:16,301::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:23:16,756::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:23:16,762::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:23:16,764::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:23:19,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:23:19,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00922233 s, 444 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:23,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:23,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102826 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:23:29,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:23:29,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874362 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:33,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:33,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807595 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:23:39,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:23:39,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776576 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:43,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:43,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138421 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:23:49,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:23:49,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872236 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:23:53,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:23:53,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866698 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:23:59,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:23:59,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908984 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:24:03,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:24:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796207 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:24:09,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:24:09,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121931 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:24:13,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:24:13,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00484575 s, 845 kB/s\n'; <rc> = 0 >Thread-14371::WARNING::2013-03-17 16:24:16,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:24:16,799::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:24:19,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:24:19,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181138 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:24:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:24:23,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757543 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:24:26,808::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:24:26,809::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:24:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:24:27,247::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:24:27,253::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:24:27,255::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:24:29,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:24:29,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018135 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:24:33,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:24:33,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941155 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:24:39,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:24:39,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101074 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:24:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:24:43,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014139 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:24:49,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:24:49,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147846 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:24:53,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:24:53,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142688 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:24:59,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:24:59,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138532 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:25:03,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:25:03,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014505 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:25:09,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:25:09,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160659 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:25:13,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:25:13,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720888 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:25:19,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:25:19,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017493 s, 234 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:25:24,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:25:24,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113708 s, 360 kB/s\n'; <rc> = 0 >Thread-14372::WARNING::2013-03-17 16:25:27,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:25:27,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:25:29,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:25:29,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945598 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:25:34,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:25:34,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209638 s, 195 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:25:37,287::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:25:37,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:25:37,722::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:25:37,728::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:25:37,730::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:25:39,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:25:39,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00795128 s, 515 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:25:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:25:44,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897873 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:25:49,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:25:50,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197645 s, 207 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:25:54,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:25:54,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082189 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:00,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:00,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149031 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:26:04,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:26:04,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852096 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:10,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:10,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743526 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:26:14,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:26:14,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130036 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:20,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:20,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713516 s, 574 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:26:24,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:26:24,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089492 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:30,152::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:26:30,154::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:26:30,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:30,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000858736 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:30,192::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000858736 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:26:30,192::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:26:30,192::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:26:30,194::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:26:30,195::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:30,196::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:30,196::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:30,196::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:30,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:30,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:30,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:30,660::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:30,661::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:26:30,663::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:26:30,664::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:26:30,665::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:26:30,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:31,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:26:31,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:31,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195302 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:26:34,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:26:34,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199835 s, 205 kB/s\n'; <rc> = 0 >Thread-14373::WARNING::2013-03-17 16:26:37,759::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:26:37,765::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:26:41,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:41,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746954 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:26:44,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:26:44,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189492 s, 216 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:26:47,768::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:26:47,769::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:26:48,235::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:26:48,242::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:26:48,243::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:26:51,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:26:51,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00564574 s, 726 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:26:54,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:26:54,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083722 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:27:01,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:27:01,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026776 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:27:04,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:27:04,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173133 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:27:11,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:27:11,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156518 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:27:14,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:27:15,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132321 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:27:21,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:27:21,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152211 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:27:25,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:27:25,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867241 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:27:31,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:27:31,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139211 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:27:35,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:27:35,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149609 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:27:41,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:27:41,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989652 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:27:45,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:27:45,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145335 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14374::WARNING::2013-03-17 16:27:48,269::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:27:48,272::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:27:51,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:27:51,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823203 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:27:55,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:27:55,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906883 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:27:58,275::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:27:58,276::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:27:58,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:27:58,738::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:27:58,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:28:01,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:28:01,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138034 s, 297 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:05,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:05,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109634 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:28:11,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:28:11,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849751 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:15,439::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:28:15,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:28:15,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:15,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0231805 s, 88.3 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:15,503::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0231805 s, 88.3 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:28:15,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:28:15,505::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:28:15,506::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:28:15,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:28:15,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:28:15,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:28:15,509::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:28:15,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:28:15,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:15,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:15,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:28:15,926::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:28:15,929::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:28:15,930::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:28:15,931::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:28:15,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:16,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:16,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:16,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00771702 s, 531 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:28:21,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:28:21,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734519 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:26,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:26,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921708 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:28:31,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:28:31,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779127 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:36,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:36,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840871 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:28:41,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:28:41,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148094 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:46,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:46,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151419 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:28:51,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:28:51,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165189 s, 248 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:28:56,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:28:56,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741466 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14375::WARNING::2013-03-17 16:28:58,785::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:28:58,787::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:29:01,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:29:01,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166762 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:29:06,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:29:06,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756137 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:29:08,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:29:08,791::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:29:09,270::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:29:09,276::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:29:09,278::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:29:11,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:29:11,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00893254 s, 459 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:29:16,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:29:16,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192611 s, 213 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:29:21,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:29:21,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013943 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:29:26,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:29:26,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139546 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:29:31,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:29:31,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147308 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:29:36,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:29:37,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858687 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:29:41,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:29:41,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887724 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:29:47,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:29:47,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838089 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:29:51,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:29:52,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168692 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:29:57,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:29:57,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775737 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:30:02,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:30:02,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096358 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:30:07,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:30:07,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00303822 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14376::WARNING::2013-03-17 16:30:09,304::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:30:09,307::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:30:12,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:30:12,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785212 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:30:17,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:30:17,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882932 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:30:19,310::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:30:19,311::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:30:19,312::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:30:19,809::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:30:19,816::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:30:19,817::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:30:22,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:30:22,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133671 s, 306 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:30:27,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:30:27,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785864 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:30:32,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:30:32,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835322 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:30:37,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:30:37,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140232 s, 292 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:30:42,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:30:42,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832382 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:30:47,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:30:47,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155335 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:30:52,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:30:52,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877138 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:30:57,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:30:57,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000679953 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:02,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:02,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155027 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:31:07,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:31:07,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143022 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:12,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:12,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017564 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:31:17,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:31:17,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103114 s, 4.0 MB/s\n'; <rc> = 0 >Thread-14377::WARNING::2013-03-17 16:31:19,850::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:31:19,854::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:31:22,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:22,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775505 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:31:27,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:31:27,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164366 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:31:29,856::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:31:29,857::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:31:30,343::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:31:30,350::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:31:30,351::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,422::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:31:32,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:31:32,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:32,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144554 s, 142 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:32,473::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144554 s, 142 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:31:32,474::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:31:32,474::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:31:32,476::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:31:32,477::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,478::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,479::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,479::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:32,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:32,866::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:31:32,867::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:31:32,869::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:31:32,871::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:31:32,871::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:31:32,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:33,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:33,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:33,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176307 s, 232 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:31:37,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:31:37,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849954 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:43,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:43,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885493 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:31:47,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:31:48,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136175 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:31:53,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:31:53,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109358 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:31:58,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:31:58,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144665 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:32:03,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:32:03,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142404 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:32:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:32:08,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846661 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:32:13,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:32:13,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000688228 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:32:18,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:32:18,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150305 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:32:23,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:32:23,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964411 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:32:28,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:32:28,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160722 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14378::WARNING::2013-03-17 16:32:30,379::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:32:30,382::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:32:33,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:32:33,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806621 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:32:38,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:32:38,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818709 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:32:40,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:32:40,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:32:40,822::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:32:40,829::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:32:40,830::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:32:43,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:32:43,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108833 s, 376 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:32:48,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:32:48,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010278 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:32:53,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:32:53,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814694 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:32:58,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:32:58,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776605 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:33:03,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:33:03,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00786566 s, 521 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:08,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:08,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762694 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:33:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:33:13,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829168 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:18,637::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:33:18,639::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:33:18,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:18,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132858 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:18,684::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132858 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:33:18,685::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:33:18,685::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:33:18,687::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:33:18,689::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:33:18,690::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:33:18,691::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:33:18,691::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:33:18,691::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:33:18,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:19,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:19,112::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:33:19,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:33:19,116::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:33:19,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:33:19,117::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:33:19,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:19,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:19,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:19,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00794957 s, 515 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:33:23,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:33:23,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105918 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:29,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:29,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208967 s, 196 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:33:33,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:33:33,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181281 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:39,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:39,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826308 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14379::WARNING::2013-03-17 16:33:40,875::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:33:40,877::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:33:43,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:33:43,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00445594 s, 919 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:33:50,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:33:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832152 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:33:50,941::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:33:50,943::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:33:51,353::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:33:51,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:33:51,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:33:53,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:33:53,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00561325 s, 730 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:34:00,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:34:00,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155158 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:34:03,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:34:03,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189099 s, 217 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:34:10,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:34:10,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762155 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:34:13,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:34:13,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795672 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:34:20,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:34:20,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864647 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:34:23,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:34:24,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118635 s, 345 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:34:30,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:34:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183365 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:34:34,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:34:34,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156435 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:34:40,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:34:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806756 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:34:44,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:34:44,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012505 s, 328 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:34:50,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:34:50,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0085828 s, 477 kB/s\n'; <rc> = 0 >Thread-14380::WARNING::2013-03-17 16:34:51,387::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:34:51,391::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:34:54,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:34:54,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108108 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:35:01,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:35:01,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02705 s, 151 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:35:01,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:35:01,395::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:35:01,821::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:35:01,828::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:35:01,830::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:35:04,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:35:04,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014603 s, 280 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:35:11,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:35:11,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875964 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:35:14,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:35:14,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159435 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:35:21,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:35:21,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100924 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:35:24,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:35:24,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014105 s, 290 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:35:31,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:35:31,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206721 s, 198 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:35:34,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:35:34,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148847 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:35:41,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:35:41,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177601 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:35:44,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:35:44,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110228 s, 372 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:35:51,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:35:51,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881877 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:35:54,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:35:54,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797448 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:36:01,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:01,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122611 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14381::WARNING::2013-03-17 16:36:01,857::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:36:01,870::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:36:04,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:04,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864758 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:36:11,892::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:36:11,893::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:36:11,894::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:11,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:12,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235646 s, 174 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:36:12,413::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:36:12,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:36:12,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:14,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100472 s, 408 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:36:22,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:22,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110712 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:24,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:24,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850951 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:36:32,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:32,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0314475 s, 130 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:34,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:36:34,657::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:36:34,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:34,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00109117 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:34,689::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00109117 s, 1.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:36:34,689::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:36:34,689::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:36:34,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:36:34,693::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:34,694::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:34,694::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:34,694::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:34,695::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:34,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:35,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:35,099::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:36:35,100::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:36:35,102::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:36:35,104::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:36:35,104::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:36:35,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:35,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:35,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:35,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151337 s, 271 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:36:42,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:42,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216566 s, 189 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:45,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:45,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135771 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:36:52,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:36:52,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787692 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:36:55,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:36:55,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145814 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:37:02,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:02,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813048 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:37:05,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:37:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00198534 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14382::WARNING::2013-03-17 16:37:12,464::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:37:12,471::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:37:12,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:12,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780304 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:37:15,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:37:15,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802086 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:37:22,482::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:37:22,484::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:22,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:22,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120028 s, 341 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:37:22,964::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:37:22,970::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:37:22,971::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:37:25,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:37:25,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106106 s, 386 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:37:32,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:32,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111425 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:37:35,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:37:35,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958595 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:37:42,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:42,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901315 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:37:45,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:37:45,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754673 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:37:52,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:37:52,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904283 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:37:55,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:37:55,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080623 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:02,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:02,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137424 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:38:05,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:38:05,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155623 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:12,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:13,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172362 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:38:15,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:38:15,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173273 s, 236 kB/s\n'; <rc> = 0 >Thread-14383::WARNING::2013-03-17 16:38:22,997::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:38:23,004::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:38:23,073::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:38:23,074::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:38:23,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:23,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000836515 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:23,101::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000836515 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:38:23,101::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:38:23,102::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:38:23,103::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:38:23,104::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:23,105::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:23,106::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:23,106::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:23,106::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:23,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:23,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:23,534::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:23,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:38:23,538::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:38:23,539::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:38:23,540::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:38:23,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:23,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:23,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:23,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316028 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:38:25,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:38:25,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164795 s, 249 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:38:33,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:38:33,008::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:38:33,414::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:38:33,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:38:33,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:38:33,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:34,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217407 s, 188 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:38:36,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:38:36,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00840123 s, 488 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:44,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:44,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111959 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:38:46,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:38:46,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146248 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:38:54,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:38:54,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160565 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:38:56,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:38:56,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163239 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:39:04,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:39:04,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129218 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:39:06,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:39:06,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769725 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:39:14,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:39:14,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818763 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:39:16,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:39:16,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162669 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:39:24,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:39:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113768 s, 360 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:39:26,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:39:26,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786937 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14384::WARNING::2013-03-17 16:39:33,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:39:33,455::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:39:34,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:39:34,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102128 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:39:36,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:39:36,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170867 s, 240 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:39:43,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:39:43,458::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:39:43,882::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:39:43,889::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:39:43,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:39:44,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:39:44,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136071 s, 301 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:39:46,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:39:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182728 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:39:54,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:39:54,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001441 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:39:56,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:39:56,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867642 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:40:04,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:40:04,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823123 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:40:06,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:40:06,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015967 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:40:14,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:40:14,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106758 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:40:16,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:40:16,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134158 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:40:24,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:40:24,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179914 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:40:26,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:40:26,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748224 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:40:34,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:40:34,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162596 s, 252 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:40:36,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:40:36,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134651 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14385::WARNING::2013-03-17 16:40:43,918::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:40:43,921::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:40:45,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:40:45,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772221 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:40:46,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:40:46,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731472 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:40:53,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:40:53,935::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:40:54,355::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:40:54,362::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:40:54,363::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:40:55,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:40:55,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013987 s, 293 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:40:56,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:40:56,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018269 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:41:05,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:41:05,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165938 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:06,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:06,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786305 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:41:15,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:41:15,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848083 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:16,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:16,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109205 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:41:25,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:41:25,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103967 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:26,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:26,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00628253 s, 652 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:41:35,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:41:35,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158486 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:36,851::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:41:36,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:41:36,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:36,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000894154 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:36,885::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000894154 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:41:36,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:41:36,886::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:41:36,888::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:41:36,889::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:41:36,890::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:41:36,890::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:41:36,893::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:41:36,894::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:41:36,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:37,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:37,316::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:41:37,317::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:41:37,319::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:41:37,321::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:41:37,321::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:41:37,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:37,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:37,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:37,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0081639 s, 502 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:41:45,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:41:45,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793811 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:47,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:47,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873515 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14386::WARNING::2013-03-17 16:41:54,389::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:41:54,393::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:41:55,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:41:55,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757008 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:41:57,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:41:57,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786918 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:42:04,394::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:42:04,396::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:42:04,397::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:42:04,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:42:04,926::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:42:04,927::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:42:05,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:42:05,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148088 s, 277 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:42:07,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:42:07,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129423 s, 316 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:42:15,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:42:15,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00856742 s, 478 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:42:17,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:42:17,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729909 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:42:25,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:42:25,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802706 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:42:27,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:42:27,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183855 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:42:35,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:42:35,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118881 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:42:37,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:42:38,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892272 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:42:46,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:42:46,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014874 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:42:48,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:42:48,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147164 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:42:56,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:42:56,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136506 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:42:58,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:42:58,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764082 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14387::WARNING::2013-03-17 16:43:04,964::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:43:04,967::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:43:06,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:06,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777483 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:43:08,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:43:08,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942344 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:43:14,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:43:14,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:43:15,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:43:15,414::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:43:15,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:16,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:16,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147665 s, 277 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:43:18,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:43:18,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127464 s, 321 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:26,372::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:43:26,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:43:26,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:26,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00105598 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:26,406::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00105598 s, 1.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:43:26,407::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:43:26,407::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:43:26,409::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:43:26,410::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:26,410::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:26,411::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:26,411::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:26,412::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:26,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:26,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:26,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:43:26,833::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:43:26,837::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:43:26,838::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:43:26,839::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:43:26,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:27,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:27,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:27,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183073 s, 224 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:43:28,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:43:28,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113895 s, 360 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:37,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:37,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874336 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:43:38,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:43:38,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129428 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:47,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:47,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719427 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:43:48,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:43:48,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977071 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:43:57,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:43:57,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144767 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:43:58,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:43:58,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116652 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:44:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:44:07,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799098 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:44:08,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:44:08,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137458 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14388::WARNING::2013-03-17 16:44:15,447::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:44:15,450::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:44:17,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:44:17,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755257 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:44:18,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:44:18,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764727 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:44:25,452::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:44:25,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:44:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:44:25,884::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:44:25,886::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:44:27,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:44:27,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111584 s, 367 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:44:28,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:44:28,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135439 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:44:37,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:44:37,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811069 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:44:38,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:44:38,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155656 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:44:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:44:47,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925889 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:44:48,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:44:48,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146093 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:44:57,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:44:57,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0240948 s, 170 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:44:58,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:44:58,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152637 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:45:08,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:45:08,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077472 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:45:08,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:45:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00330763 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:45:18,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:45:18,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794437 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:45:18,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:45:18,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135536 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14389::WARNING::2013-03-17 16:45:25,922::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:45:25,925::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:45:28,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:45:28,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076414 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:45:28,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:45:28,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165323 s, 248 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:45:35,931::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:45:35,939::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:45:36,399::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:45:36,405::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:45:36,407::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:45:38,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:45:38,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204759 s, 200 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:45:38,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:45:38,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170655 s, 240 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:45:48,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:45:48,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208003 s, 197 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:45:48,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:45:48,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155139 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:45:58,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:45:58,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201553 s, 203 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:45:58,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:45:58,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138196 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:46:08,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:46:08,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185065 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:08,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:09,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099101 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:46:18,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:46:18,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150862 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:19,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:19,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112978 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:46:28,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:46:28,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858283 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:29,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:29,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138871 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14390::WARNING::2013-03-17 16:46:36,433::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:46:36,437::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:46:38,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:46:38,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859513 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:39,191::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:46:39,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:46:39,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:39,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0007951 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:39,224::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0007951 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:46:39,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:46:39,224::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:46:39,226::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:46:39,227::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:46:39,228::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:46:39,228::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:46:39,229::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:46:39,229::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:46:39,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:39,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:39,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:46:39,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:46:39,667::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:46:39,669::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:46:39,669::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:46:39,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:40,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:40,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:40,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124397 s, 329 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:46:46,439::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:46:46,440::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:46:46,894::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:46:46,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:46:46,902::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:46:48,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:46:48,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144631 s, 283 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:46:50,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:46:50,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163553 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:46:58,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:46:59,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750729 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:47:00,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:47:00,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175712 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:47:09,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:47:09,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224193 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:47:10,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:47:10,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018773 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:47:19,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:47:19,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701189 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:47:20,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:47:20,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783784 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:47:29,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:47:29,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198655 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:47:30,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:47:30,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181371 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:47:39,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:47:39,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749288 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:47:40,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:47:40,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015301 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14391::WARNING::2013-03-17 16:47:46,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:47:46,933::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 16:47:49,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:47:49,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076623 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:47:50,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:47:50,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837816 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:47:56,942::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:47:56,943::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:47:56,945::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:47:57,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:47:57,421::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:47:57,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:47:59,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:47:59,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140799 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:48:00,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:00,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128349 s, 319 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:09,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:09,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151465 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:48:10,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:10,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101136 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:19,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:19,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740613 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:48:20,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:20,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759458 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:29,702::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:48:29,704::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:48:29,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:29,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133887 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:29,741::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133887 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:48:29,742::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:48:29,742::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:48:29,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:48:29,745::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:48:29,745::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:48:29,746::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:48:29,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:48:29,746::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:48:29,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:30,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:30,174::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:48:30,176::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:48:30,178::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:48:30,179::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:48:30,180::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:48:30,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:30,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:30,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:30,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:30,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152442 s, 269 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:30,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108562 s, 377 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:48:40,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:40,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142173 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:40,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:40,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184753 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:48:50,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:48:50,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146314 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:48:50,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:48:50,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147533 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14392::WARNING::2013-03-17 16:48:57,453::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:48:57,459::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:49:00,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:49:00,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859834 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:49:00,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:49:00,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069084 s, 5.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:49:07,463::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:49:07,464::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:49:07,925::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:49:07,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:49:07,933::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:49:10,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:49:10,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00827356 s, 495 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:49:10,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:49:10,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130319 s, 314 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:49:20,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:49:20,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748295 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:49:21,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:49:21,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718671 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:49:30,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:49:30,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902909 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:49:31,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:49:31,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154189 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:49:40,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:49:40,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081235 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:49:41,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:49:41,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082438 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:49:50,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:49:50,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152318 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:49:51,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:49:51,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017058 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:50:01,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:50:01,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144336 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:50:01,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:50:01,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012241 s, 335 kB/s\n'; <rc> = 0 >Thread-14393::WARNING::2013-03-17 16:50:07,961::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:50:07,967::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:50:11,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:50:11,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109803 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:50:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:50:11,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194604 s, 210 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:50:17,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:50:17,971::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:50:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:50:18,552::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:50:18,553::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:50:21,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:50:21,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223809 s, 183 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:50:21,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:50:21,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185283 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:50:31,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:50:31,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124935 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:50:31,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:50:31,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184483 s, 222 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:50:41,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:50:41,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:50:41,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:50:41,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017636 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:50:51,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:50:51,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169258 s, 242 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:50:51,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:50:51,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841707 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:01,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:01,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251277 s, 163 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:51:02,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:02,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146065 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:11,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:11,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137513 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:51:12,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:12,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153165 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14394::WARNING::2013-03-17 16:51:18,579::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:51:18,583::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:51:21,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:21,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886793 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:51:22,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:22,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906977 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:51:28,588::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:51:28,590::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:51:29,068::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:51:29,074::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:51:29,076::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:31,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:31,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155436 s, 264 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:51:32,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:32,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0347193 s, 118 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:41,520::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:51:41,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:51:41,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:41,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000911044 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:41,555::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000911044 s, 2.2 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:51:41,556::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:51:41,556::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:51:41,557::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:51:41,559::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:41,559::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:41,560::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:41,560::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:41,560::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:41,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:42,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:42,005::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:51:42,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:51:42,009::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:51:42,010::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:51:42,010::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:51:42,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:42,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:42,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:42,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122804 s, 334 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:42,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154205 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:51:52,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:51:52,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:51:52,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114568 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:51:52,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860732 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:52:02,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:52:02,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:52:02,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749896 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:52:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693911 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:52:12,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:52:12,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766568 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:52:12,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:52:12,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235334 s, 174 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:52:22,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:52:22,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151239 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:52:22,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:52:22,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146058 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14395::WARNING::2013-03-17 16:52:29,116::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:52:29,120::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:52:32,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:52:32,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159358 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:52:32,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:52:32,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824983 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:52:39,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:52:39,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:52:39,569::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:52:39,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:52:39,577::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:52:42,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:52:42,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146137 s, 280 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:52:43,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:52:43,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017374 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:52:52,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:52:52,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791549 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:52:53,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:52:53,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228339 s, 179 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:53:02,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:53:02,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203943 s, 201 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:03,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:03,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156278 s, 262 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:53:12,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:53:13,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154935 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:13,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:13,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169494 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:53:23,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:53:23,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754606 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:23,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849474 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:53:33,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:53:33,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131853 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:33,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:53:33,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:53:33,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:33,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00158107 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:33,504::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00158107 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:53:33,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:53:33,505::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:53:33,507::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:53:33,508::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:53:33,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:53:33,509::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:53:33,509::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:53:33,510::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:53:33,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:33,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:33,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:53:33,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:53:33,928::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:53:33,929::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:53:33,930::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:53:33,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:34,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:34,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:34,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00499275 s, 820 kB/s\n'; <rc> = 0 >Thread-14396::WARNING::2013-03-17 16:53:39,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:53:39,609::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:53:43,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:53:43,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767993 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:44,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:44,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306399 s, 134 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:53:49,611::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:53:49,612::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:53:49,613::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:53:50,057::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:53:50,064::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:53:50,065::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:53:53,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:53:53,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121283 s, 338 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:53:54,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:53:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161034 s, 254 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:54:03,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:54:03,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802093 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:54:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:54:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187094 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:54:13,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:54:13,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827549 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:54:14,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:54:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136328 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:54:23,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:54:23,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699746 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:54:24,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:54:24,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193144 s, 212 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:54:33,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:54:33,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217798 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:54:34,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:54:34,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148044 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:54:43,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:54:43,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806421 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:54:44,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:54:44,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177881 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14397::WARNING::2013-03-17 16:54:50,127::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:54:50,142::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:54:53,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:54:53,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122929 s, 333 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:54:55,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:54:55,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749041 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:55:00,145::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:55:00,146::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:55:00,639::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:55:00,645::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:55:00,647::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:55:03,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:55:03,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947918 s, 432 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:55:05,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:55:05,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801408 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:55:13,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:55:13,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165004 s, 248 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:55:15,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:55:15,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877039 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:55:23,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:55:23,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00310029 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:55:25,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:55:25,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102952 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:55:33,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:55:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149367 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:55:35,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:55:35,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847625 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:55:43,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:55:43,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912045 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:55:45,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:55:45,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140011 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:55:53,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:55:53,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166232 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:55:55,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:55:55,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183806 s, 223 kB/s\n'; <rc> = 0 >Thread-14398::WARNING::2013-03-17 16:56:00,675::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:56:00,678::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:56:03,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:03,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836947 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:56:05,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:56:05,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078648 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:56:10,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:56:10,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:56:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:56:11,193::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:56:11,195::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:13,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:13,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204397 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:56:15,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:56:15,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.004725 s, 867 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:23,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:23,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860118 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:56:25,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:56:25,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765874 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:33,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:33,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147439 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:56:35,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:56:35,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691983 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:43,928::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 16:56:43,929::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:56:43,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:43,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000732418 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:43,963::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000732418 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 16:56:43,964::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 16:56:43,964::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 16:56:43,967::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 16:56:43,968::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:43,969::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:43,969::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:43,970::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:43,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:43,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:44,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:44,426::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:56:44,427::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 16:56:44,429::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 16:56:44,430::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 16:56:44,431::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 16:56:44,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:44,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:44,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:44,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188054 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:56:45,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:56:45,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140986 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:56:54,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:56:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140033 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:56:55,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:56:55,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150493 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:57:04,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:57:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836608 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:57:06,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:57:06,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00468876 s, 874 kB/s\n'; <rc> = 0 >Thread-14399::WARNING::2013-03-17 16:57:11,220::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:57:11,224::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:57:14,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:57:14,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167926 s, 244 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:57:16,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:57:16,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139446 s, 2.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:57:21,226::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:57:21,227::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:57:21,722::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:57:21,729::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:57:21,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:57:24,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:57:25,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103171 s, 397 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:57:26,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:57:26,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015475 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:57:35,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:57:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834459 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:57:36,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:57:36,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950387 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:57:45,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:57:45,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859351 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:57:46,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:57:46,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105354 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:57:55,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:57:55,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811168 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:57:56,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:57:56,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108554 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:58:05,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:58:05,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095618 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:06,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:06,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149216 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:58:15,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:58:15,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748903 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:16,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:16,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153583 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14400::WARNING::2013-03-17 16:58:21,797::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:58:21,800::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:58:25,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:58:25,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179839 s, 228 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:26,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:26,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926994 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:58:31,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:58:31,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:58:32,261::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:58:32,268::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:58:32,269::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:58:35,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:58:35,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148933 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:36,720::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 16:58:36,722::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:58:36,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:36,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000816587 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:36,748::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000816587 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 16:58:36,748::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 16:58:36,748::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 16:58:36,750::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 16:58:36,751::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:58:36,752::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:58:36,752::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:58:36,752::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:58:36,753::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 16:58:36,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:37,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:37,167::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 16:58:37,168::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 16:58:37,170::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 16:58:37,171::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 16:58:37,172::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 16:58:37,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:37,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:37,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101477 s, 404 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:58:45,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:58:45,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880119 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:47,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:47,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014529 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:58:55,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:58:55,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899506 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:58:57,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:58:57,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861735 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:59:05,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:59:05,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103311 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:59:07,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:59:07,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145455 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:59:15,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:59:15,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831667 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:59:17,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:59:17,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839102 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:59:25,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:59:25,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101976 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:59:27,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:59:27,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215833 s, 190 kB/s\n'; <rc> = 0 >Thread-14401::WARNING::2013-03-17 16:59:32,332::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 16:59:32,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 16:59:35,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:59:35,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172157 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:59:38,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:59:38,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790188 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 16:59:42,341::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 16:59:42,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 16:59:42,343::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 16:59:42,967::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 16:59:42,973::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 16:59:42,974::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 16:59:45,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:59:45,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0464167 s, 88.2 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:59:48,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:59:48,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144414 s, 284 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 16:59:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 16:59:55,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182877 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 16:59:58,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 16:59:58,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750748 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:00:05,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:00:05,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132343 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:00:08,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:00:08,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786907 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:00:15,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:00:15,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812442 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:00:18,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:00:18,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204675 s, 200 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:00:25,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:00:25,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143374 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:00:28,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:00:28,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154277 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:00:35,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:00:35,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191274 s, 214 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:00:38,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:00:38,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183154 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14402::WARNING::2013-03-17 17:00:43,020::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:00:43,023::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:00:45,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:00:45,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766254 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:00:48,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:00:48,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780549 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:00:53,030::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:00:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:00:53,486::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:00:53,493::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:00:53,494::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:00:55,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:00:55,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123413 s, 332 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:00:58,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:00:58,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00779426 s, 526 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:06,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:06,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019987 s, 205 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:01:08,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:01:08,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179795 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:16,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:16,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815591 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:01:18,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:01:18,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150555 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:26,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:26,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196843 s, 208 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:01:28,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:01:28,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185666 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:36,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:36,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130087 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:01:38,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:01:39,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797838 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:46,199::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:01:46,200::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:01:46,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:46,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0216253 s, 94.7 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:46,272::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0216253 s, 94.7 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:01:46,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:01:46,273::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:01:46,274::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:01:46,275::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:01:46,281::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:01:46,281::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:01:46,282::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:01:46,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:01:46,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:46,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:46,710::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:01:46,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:01:46,714::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:01:46,715::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:01:46,716::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:01:46,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:47,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:01:47,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:47,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128843 s, 318 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:01:49,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:01:49,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00915144 s, 448 kB/s\n'; <rc> = 0 >Thread-14403::WARNING::2013-03-17 17:01:53,526::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:01:53,530::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:01:57,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:01:57,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862414 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:01:59,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:01:59,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929631 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:02:03,536::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:02:03,537::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:02:04,020::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:02:04,026::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:02:04,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:02:07,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:02:07,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161589 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:02:09,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:02:09,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014275 s, 287 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:02:17,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:02:17,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223871 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:02:19,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:02:19,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152576 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:02:27,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:02:27,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149264 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:02:29,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:02:29,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173734 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:02:37,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:02:37,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167159 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:02:39,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:02:39,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00698091 s, 587 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:02:47,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:02:47,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172293 s, 238 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:02:49,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:02:49,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183485 s, 223 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:02:57,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:02:57,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166709 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:02:59,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:02:59,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123346 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14404::WARNING::2013-03-17 17:03:04,065::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:03:04,071::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:03:07,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:03:07,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870279 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:09,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:09,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078894 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:03:14,074::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:03:14,075::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:03:14,523::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:03:14,530::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:03:14,531::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:03:17,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:03:17,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110308 s, 371 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:19,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:19,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157987 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:03:27,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:03:27,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230888 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:29,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:29,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845866 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:03:37,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:03:37,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815422 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:39,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:03:39,986::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:03:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:40,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130855 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:40,031::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130855 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:03:40,032::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:03:40,032::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:03:40,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:03:40,035::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:03:40,035::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:03:40,036::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:03:40,036::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:03:40,037::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:03:40,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:40,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:40,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:03:40,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:03:40,470::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:03:40,471::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:03:40,472::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:03:40,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:40,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:40,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:40,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115221 s, 355 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:03:47,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:03:47,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155022 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:03:50,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:03:50,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150409 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:03:57,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:03:57,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141982 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:04:01,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:04:01,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758659 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:04:07,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:04:07,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855992 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:04:11,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:04:11,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142657 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14405::WARNING::2013-03-17 17:04:14,557::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:04:14,560::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:04:17,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:04:17,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881536 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:04:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:04:21,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100679 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:04:24,573::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:04:24,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:04:25,024::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:04:25,030::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:04:25,032::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:04:27,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:04:27,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122609 s, 334 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:04:31,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:04:31,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837562 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:04:37,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:04:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110686 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:04:41,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:04:41,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786645 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:04:47,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:04:47,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161242 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:04:51,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:04:51,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117896 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:04:57,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:04:57,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146597 s, 279 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:05:01,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:05:01,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165476 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:05:07,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:05:07,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136376 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:05:11,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:05:11,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188009 s, 218 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:05:18,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:05:18,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156011 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:05:21,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:05:21,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821616 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14406::WARNING::2013-03-17 17:05:25,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:05:25,066::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:05:28,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:05:28,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760369 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:05:31,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:05:31,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158476 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:05:35,068::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:05:35,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:05:35,070::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:05:35,504::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:05:35,511::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:05:35,512::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:05:38,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:05:38,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138563 s, 296 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:05:41,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:05:41,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784211 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:05:48,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:05:48,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763205 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:05:51,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:05:51,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100574 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:05:58,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:05:58,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163374 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:06:01,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:06:02,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762675 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:08,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:08,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909693 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:06:12,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:06:12,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154455 s, 265 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:18,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:18,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124002 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:06:22,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:06:22,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147293 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:28,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:28,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767112 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:06:32,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:06:32,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154934 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14407::WARNING::2013-03-17 17:06:35,538::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:06:35,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:06:38,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:38,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810725 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:06:42,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:06:42,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821592 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:06:45,554::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:06:45,555::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:06:46,019::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:06:46,025::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:06:46,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,409::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:06:48,413::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:06:48,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:48,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0152787 s, 134 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:48,460::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0152787 s, 134 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:06:48,461::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:06:48,461::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:06:48,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:06:48,466::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,467::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,467::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,468::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,468::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:48,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:48,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:06:48,901::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:06:48,904::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:06:48,905::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:06:48,906::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:06:48,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:49,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:49,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:49,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00411293 s, 996 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:06:52,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:06:52,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146846 s, 279 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:06:59,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:06:59,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164381 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:07:02,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:07:02,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835919 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:07:09,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:07:09,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169705 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:07:12,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:07:12,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828625 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:07:19,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:07:19,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140053 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:07:22,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:07:22,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077726 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:07:29,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:07:29,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086658 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:07:32,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:07:32,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162638 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:07:39,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:07:39,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124354 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:07:42,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:07:42,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166477 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14408::WARNING::2013-03-17 17:07:46,055::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:07:46,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:07:49,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:07:49,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194686 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:07:52,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:07:52,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076887 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:07:56,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:07:56,062::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:07:56,483::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:07:56,490::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:07:56,491::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:07:59,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:07:59,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202571 s, 202 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:02,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:02,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805248 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:08:09,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:08:09,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799148 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:12,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:13,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831745 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:08:19,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:08:19,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215665 s, 190 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:23,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:23,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135964 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:08:29,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:08:29,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163236 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:33,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:33,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149241 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:08:39,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:08:39,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132257 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:43,211::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:08:43,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:08:43,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:43,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148936 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:43,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148936 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:08:43,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:08:43,260::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:08:43,262::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:08:43,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:08:43,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:08:43,264::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:08:43,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:08:43,265::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:08:43,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:43,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:43,709::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:08:43,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:08:43,713::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:08:43,714::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:08:43,715::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:08:43,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:44,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:44,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:44,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0065085 s, 629 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:08:49,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:08:49,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138121 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:08:54,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:08:54,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955861 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14409::WARNING::2013-03-17 17:08:56,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:08:56,526::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:08:59,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:08:59,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860797 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:09:04,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:09:04,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185312 s, 221 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:09:06,530::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:09:06,531::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:09:07,111::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:09:07,118::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:09:07,120::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:09:09,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:09:09,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150959 s, 271 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:09:14,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:09:14,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837921 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:09:20,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:09:20,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074549 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:09:24,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:09:24,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104575 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:09:30,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:09:30,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089316 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:09:34,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:09:34,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813745 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:09:40,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:09:40,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699955 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:09:44,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:09:44,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833623 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:09:50,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:09:50,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00661668 s, 619 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:09:54,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:09:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784417 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:10:00,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:10:00,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168491 s, 243 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:10:04,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:10:04,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110655 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14410::WARNING::2013-03-17 17:10:07,146::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:10:07,152::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:10:10,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:10:10,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845291 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:10:14,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:10:14,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101787 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:10:17,154::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:10:17,155::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:10:17,637::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:10:17,644::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:10:17,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:10:20,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:10:20,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165621 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:10:24,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:10:24,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923208 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:10:30,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:10:30,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789782 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:10:34,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:10:34,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00657933 s, 623 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:10:40,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:10:40,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143028 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:10:45,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:10:45,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167387 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:10:50,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:10:50,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144097 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:10:55,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:10:55,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893258 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:00,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:00,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148989 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:11:05,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:11:05,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189473 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:10,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:10,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134843 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:11:15,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:11:15,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207146 s, 198 kB/s\n'; <rc> = 0 >Thread-14411::WARNING::2013-03-17 17:11:17,673::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:11:17,677::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:11:20,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:20,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00653537 s, 627 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:11:25,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:11:25,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373736 s, 1.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:11:27,680::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:11:27,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:11:27,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:11:28,144::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:11:28,150::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:11:28,152::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:30,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102099 s, 401 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:11:35,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:11:35,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197217 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:40,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001244 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:11:45,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:11:45,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175203 s, 234 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:50,820::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:11:50,821::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:11:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:50,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00486282 s, 421 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:50,858::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00486282 s, 421 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:11:50,860::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:11:50,861::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:11:50,862::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:11:50,864::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:50,864::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:50,865::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:50,865::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:50,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:51,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:51,307::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:11:51,308::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:11:51,311::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:11:51,312::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:11:51,313::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:11:51,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:51,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:11:51,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:11:51,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159297 s, 257 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:11:55,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:11:55,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148057 s, 277 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:12:01,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:12:01,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779641 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:12:05,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:12:05,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147853 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:12:11,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:12:11,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749236 s, 547 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:12:15,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:12:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137795 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:12:21,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:12:21,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170359 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:12:25,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:12:25,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733714 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14412::WARNING::2013-03-17 17:12:28,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:12:28,185::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:12:31,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:12:31,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008029 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:12:36,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:12:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821853 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:12:38,189::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:12:38,190::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:12:38,648::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:12:38,655::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:12:38,656::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:12:41,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:12:42,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166119 s, 247 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:12:46,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:12:46,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079972 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:12:52,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:12:52,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828348 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:12:56,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:12:56,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201474 s, 203 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:13:02,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:13:02,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816155 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:06,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:06,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868781 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:13:12,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:13:12,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100896 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:16,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:16,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768662 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:13:22,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:13:22,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176447 s, 232 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:26,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:26,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144119 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:13:32,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:13:32,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00447181 s, 916 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:36,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:36,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894619 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14413::WARNING::2013-03-17 17:13:38,681::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:13:38,684::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:13:42,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:13:42,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200953 s, 204 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:46,564::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:13:46,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:13:46,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:46,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000789984 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:46,594::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000789984 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:13:46,594::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:13:46,594::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:13:46,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:13:46,598::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:13:46,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:13:46,599::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:13:46,600::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:13:46,600::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:13:46,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:47,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:47,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:13:47,029::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:13:47,032::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:13:47,033::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:13:47,034::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:13:47,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:47,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:47,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:47,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152789 s, 268 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:13:48,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:13:48,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:13:49,178::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:13:49,185::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:13:49,186::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:13:52,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:13:52,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107902 s, 380 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:13:57,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:13:57,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854245 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:14:02,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:14:02,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810652 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:14:07,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:14:07,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0020756 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:14:12,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:14:12,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131632 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:14:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:14:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204147 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:14:22,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:14:22,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754601 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:14:27,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:14:27,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897578 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:14:32,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:14:32,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154153 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:14:37,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:14:37,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00719033 s, 570 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:14:42,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:14:42,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104742 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:14:47,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:14:48,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203467 s, 201 kB/s\n'; <rc> = 0 >Thread-14414::WARNING::2013-03-17 17:14:49,215::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:14:49,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:14:52,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:14:52,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789801 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:14:58,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:14:58,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796018 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:14:59,220::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:14:59,221::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:14:59,659::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:14:59,666::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:14:59,667::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:15:02,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:15:02,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222691 s, 184 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:15:08,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:15:08,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773591 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:15:12,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:15:12,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080348 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:15:18,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:15:18,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835142 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:15:22,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:15:22,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015816 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:15:28,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:15:28,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373321 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:15:32,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:15:32,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013937 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:15:38,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:15:38,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000665336 s, 6.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:15:42,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:15:42,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135915 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:15:48,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:15:48,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834516 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:15:52,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:15:52,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780389 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:15:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:15:59,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725391 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14415::WARNING::2013-03-17 17:15:59,697::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:15:59,700::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:16:02,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:02,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018645 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:16:09,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:16:09,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135889 s, 3.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:16:09,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:16:09,704::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:16:10,190::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:16:10,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:16:10,198::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:12,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:13,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115214 s, 356 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:16:19,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:16:19,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879443 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:23,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:23,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294203 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:16:29,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:16:29,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817113 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:33,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:33,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850531 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:16:39,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:16:39,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078003 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:43,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:43,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915663 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:16:49,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:16:49,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016636 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:53,134::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:16:53,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:16:53,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:53,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136822 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:53,174::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136822 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:16:53,175::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:16:53,175::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:16:53,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:16:53,178::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:53,178::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:53,179::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:53,179::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:53,179::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:53,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:53,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:53,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:16:53,599::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:16:53,603::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:16:53,604::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:16:53,604::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:16:53,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:53,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:16:53,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:16:54,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166361 s, 246 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:16:59,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:16:59,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133919 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:17:04,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:17:04,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142844 s, 287 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:17:09,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:17:09,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00848529 s, 483 kB/s\n'; <rc> = 0 >Thread-14416::WARNING::2013-03-17 17:17:10,225::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:17:10,235::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:17:14,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:17:14,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903263 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:17:20,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 17:17:20,295::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:17:20,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:17:20,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 17:17:20,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.028989 s, 141 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:17:20,753::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:17:20,758::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:17:20,760::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:17:24,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:17:24,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182437 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:17:30,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:17:30,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101292 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:17:34,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:17:34,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000686609 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:17:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:17:40,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126004 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:17:44,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:17:44,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105803 s, 387 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:17:50,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:17:50,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170649 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:17:54,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:17:54,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010417 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:00,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:00,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199113 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:18:04,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:18:04,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0317945 s, 129 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:10,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:10,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185044 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:18:14,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:18:14,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999192 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14417::WARNING::2013-03-17 17:18:20,787::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:18:20,793::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:18:20,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:20,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119444 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:18:24,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:18:24,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106142 s, 386 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:18:30,796::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:18:30,797::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:30,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:31,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121827 s, 336 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:18:31,300::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:18:31,306::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:18:31,307::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:18:34,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:18:34,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013015 s, 315 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:41,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:41,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911369 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:18:44,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:18:44,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017535 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:51,116::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:18:51,118::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:18:51,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:51,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000784722 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:51,151::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000784722 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:18:51,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:18:51,152::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:18:51,153::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:18:51,154::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:18:51,155::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:18:51,155::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:18:51,156::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:18:51,156::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:18:51,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:51,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:51,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:18:51,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:18:51,599::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:18:51,600::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:18:51,601::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:18:51,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:52,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:18:52,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:18:52,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013852 s, 296 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:18:54,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:18:54,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120981 s, 339 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:19:02,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:19:02,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199102 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:19:04,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:19:04,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748276 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:19:12,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:19:12,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199518 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:19:14,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:19:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140647 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:19:22,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:19:22,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01789 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:19:24,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:19:24,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169828 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14418::WARNING::2013-03-17 17:19:31,337::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:19:31,343::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:19:32,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:19:32,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875983 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:19:34,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:19:34,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834385 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:19:41,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:19:41,357::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:19:41,749::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:19:41,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:19:41,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:19:42,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:19:42,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178916 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:19:44,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:19:44,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187043 s, 219 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:19:52,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:19:52,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143531 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:19:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:19:54,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137025 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:20:02,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:20:02,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863357 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:20:04,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:20:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015758 s, 260 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:20:12,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:20:12,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816165 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:20:14,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:20:14,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199892 s, 205 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:20:22,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:20:22,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016561 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:20:25,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:20:25,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147093 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:20:32,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:20:32,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140417 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:20:35,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:20:35,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801976 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14419::WARNING::2013-03-17 17:20:41,782::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:20:41,786::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:20:42,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:20:42,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778921 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:20:45,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:20:45,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168833 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:20:51,794::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:20:51,795::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:20:52,313::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:20:52,319::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:20:52,321::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:20:52,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:20:53,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121346 s, 338 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:20:55,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:20:55,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132706 s, 309 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:21:03,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:21:03,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166573 s, 246 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:05,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:05,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170054 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:21:13,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:21:13,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226051 s, 181 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:15,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:15,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094066 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:21:23,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:21:23,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806264 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:25,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:25,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972916 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:21:33,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:21:33,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161427 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:35,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:35,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761918 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:21:43,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:21:43,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823785 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:45,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:45,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121224 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14420::WARNING::2013-03-17 17:21:52,345::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:21:52,349::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:21:53,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:21:53,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808664 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:55,401::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:21:55,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:21:55,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:55,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000868357 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:55,430::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000868357 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:21:55,431::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:21:55,431::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:21:55,433::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:21:55,434::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:21:55,434::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:21:55,435::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:21:55,435::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:21:55,435::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:21:55,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:55,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:55,863::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:21:55,864::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:21:55,866::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:21:55,867::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:21:55,868::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:21:55,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:56,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:21:56,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:21:56,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153391 s, 267 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:22:02,351::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:22:02,353::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:22:02,866::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:22:02,873::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:22:02,874::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:22:03,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:22:03,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011323 s, 362 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:22:06,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:22:06,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122877 s, 333 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:22:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:22:13,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894641 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:22:16,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:22:16,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793507 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:22:23,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:22:23,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175234 s, 234 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:22:26,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:22:26,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811971 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:22:33,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:22:33,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101213 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:22:36,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:22:36,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144253 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:22:43,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:22:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171672 s, 239 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:22:46,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:22:46,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147315 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:22:53,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:22:54,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140992 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:22:56,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:22:56,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288663 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14421::WARNING::2013-03-17 17:23:02,901::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:23:02,905::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:23:04,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:04,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111831 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:23:06,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:23:06,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720115 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:23:12,907::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:23:12,909::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:23:12,910::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:23:13,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:23:13,424::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:23:13,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:14,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:14,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138024 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:23:16,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:23:16,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123516 s, 332 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:24,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:24,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750304 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:23:26,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:23:26,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179034 s, 229 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:34,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:34,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101316 s, 404 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:23:36,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:23:36,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191783 s, 214 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:44,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:44,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00945544 s, 433 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:23:46,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:23:46,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174581 s, 235 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:54,482::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:23:54,483::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:23:54,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:54,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0150077 s, 136 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:54,544::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0150077 s, 136 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:23:54,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:23:54,546::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:23:54,547::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:23:54,548::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:54,549::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:54,549::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:54,550::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:54,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:54,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:54,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:54,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:23:54,983::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:23:54,985::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:23:54,986::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:23:54,987::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:23:54,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:55,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:23:55,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:23:55,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586801 s, 698 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:23:56,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:23:56,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163273 s, 251 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:24:05,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:24:05,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838938 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:24:06,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:24:06,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136021 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14422::WARNING::2013-03-17 17:24:13,457::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:24:13,461::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:24:15,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:24:15,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801398 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:24:16,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:24:16,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106741 s, 3.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:24:23,471::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:24:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:24:23,922::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:24:23,929::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:24:23,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:24:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:24:25,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017716 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:24:26,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:24:26,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014358 s, 285 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:24:35,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:24:35,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018043 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:24:36,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:24:37,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799585 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:24:45,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:24:45,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793601 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:24:47,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:24:47,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814068 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:24:55,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:24:55,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019727 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:24:57,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:24:57,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222172 s, 184 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:25:05,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:25:05,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883545 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:25:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:25:07,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322402 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:25:16,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:25:16,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816469 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:25:17,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:25:17,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123256 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14423::WARNING::2013-03-17 17:25:23,955::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:25:23,961::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:25:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:25:26,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899746 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:25:27,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:25:27,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764851 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:25:33,964::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:25:33,966::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:25:34,439::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:25:34,445::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:25:34,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:25:36,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:25:36,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193205 s, 212 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:25:37,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:25:37,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192225 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:25:46,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:25:46,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780071 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:25:47,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:25:47,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083969 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:25:56,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:25:56,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113691 s, 360 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:25:57,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:25:57,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956623 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:26:06,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:26:06,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088409 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:07,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:07,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173279 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:26:16,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:26:16,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191539 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:17,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:17,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087147 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:26:26,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:26:26,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218718 s, 187 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:27,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:27,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125513 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14424::WARNING::2013-03-17 17:26:34,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:26:34,474::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:26:36,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:26:36,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085778 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:37,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:37,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821791 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:26:44,479::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:26:44,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:26:44,995::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:26:45,001::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:26:45,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:26:46,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:26:46,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114009 s, 359 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:47,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:47,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160628 s, 255 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:26:56,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:26:56,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841187 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:57,592::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:26:57,593::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:26:57,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:57,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00322288 s, 635 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:57,630::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00322288 s, 635 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:26:57,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:26:57,631::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:26:57,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:26:57,634::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:26:57,634::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:26:57,635::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:26:57,635::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:26:57,636::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:26:57,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:58,063::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:26:58,064::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:26:58,068::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:26:58,069::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:26:58,069::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:26:58,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:58,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:26:58,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:26:58,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140389 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:27:06,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:27:06,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840767 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:27:08,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:27:08,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146463 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:27:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:27:17,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014893 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:27:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:27:18,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143248 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:27:27,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:27:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119143 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:27:28,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:27:28,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074809 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:27:37,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:27:37,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133859 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:27:38,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:27:38,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215223 s, 190 kB/s\n'; <rc> = 0 >Thread-14425::WARNING::2013-03-17 17:27:45,030::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:27:45,033::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:27:47,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:27:47,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770944 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:27:48,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:27:48,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180842 s, 226 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:27:55,036::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:27:55,037::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:27:55,522::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:27:55,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:27:55,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:27:57,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:27:57,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098911 s, 414 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:27:58,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:27:58,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171225 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:07,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:07,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744008 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:28:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:28:08,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979771 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:17,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:17,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804456 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:28:18,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:28:18,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160244 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:27,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:27,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176217 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:28:28,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:28:28,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134912 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:37,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:37,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159167 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:28:38,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:28:38,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139803 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:47,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:47,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192117 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:28:48,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:28:49,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149591 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14426::WARNING::2013-03-17 17:28:55,564::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:28:55,568::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:28:57,809::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:28:57,810::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:28:57,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:57,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00085227 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:57,838::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00085227 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:28:57,838::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:28:57,838::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:28:57,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:28:57,841::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:28:57,842::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:28:57,842::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:28:57,842::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:28:57,843::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:28:57,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:58,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:58,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:28:58,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:28:58,451::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:28:58,452::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:28:58,452::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:28:58,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:28:58,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:28:58,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245075 s, 167 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:28:59,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:28:59,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100839 s, 406 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:29:05,573::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:29:05,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:29:05,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:29:06,014::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:29:06,020::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:29:06,022::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:29:08,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:29:08,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191644 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:29:09,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:29:09,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164528 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:29:18,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:29:19,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805517 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:29:19,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:29:19,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917674 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:29:29,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:29:29,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001706 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:29:29,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:29:29,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874796 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:29:39,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:29:39,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839252 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:29:39,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:29:39,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109592 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:29:49,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:29:49,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145186 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:29:49,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:29:49,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116114 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:29:59,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:29:59,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790661 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:29:59,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:29:59,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753901 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14427::WARNING::2013-03-17 17:30:06,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:30:06,053::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:30:09,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:09,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838735 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:09,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:30:09,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088964 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:30:16,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:30:16,057::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:30:16,589::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:30:16,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:30:16,597::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:30:19,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:19,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116285 s, 352 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:30:19,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145374 s, 282 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:29,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:29,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:29,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011338 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:29,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148898 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:39,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:30:39,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778719 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:30:39,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:39,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00506969 s, 808 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:49,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:30:49,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777059 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:30:49,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:49,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947272 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:30:59,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:30:59,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169612 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:30:59,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:30:59,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011329 s, 362 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:31:09,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:31:09,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141632 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:31:09,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:31:10,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009583 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14428::WARNING::2013-03-17 17:31:16,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:31:16,630::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:31:19,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:31:19,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757862 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:31:20,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:31:20,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144685 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:31:26,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:31:26,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:31:27,082::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:31:27,089::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:31:27,090::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:31:29,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:31:29,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807738 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:31:30,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:31:30,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00805217 s, 509 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:31:39,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:31:39,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867986 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:31:40,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:31:40,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932245 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:31:49,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:31:49,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801162 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:31:50,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:31:50,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134253 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:31:59,998::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:31:59,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:32:00,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:00,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000858452 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:00,031::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000858452 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:32:00,031::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:32:00,032::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:32:00,033::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:32:00,035::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:32:00,035::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:32:00,036::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:32:00,036::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:32:00,036::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:32:00,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:00,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:00,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222663 s, 184 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:00,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:00,485::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:32:00,486::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:32:00,501::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:32:00,503::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:32:00,505::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:32:00,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:00,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:00,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:00,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118128 s, 347 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:32:10,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:10,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216731 s, 189 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:10,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:10,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015339 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:32:20,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:20,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900507 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:21,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:21,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879758 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14429::WARNING::2013-03-17 17:32:27,119::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:32:27,128::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:32:30,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:30,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786645 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:31,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867842 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:32:37,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:32:37,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:32:37,649::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:32:37,655::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:32:37,657::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:32:40,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:40,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194686 s, 210 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:41,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:41,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183102 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:32:50,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:32:50,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143504 s, 285 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:32:51,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:32:51,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150647 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:33:00,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:33:01,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861436 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:33:01,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:33:01,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851762 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:33:11,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:33:11,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179244 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:33:11,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:33:11,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157049 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:33:21,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:33:21,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160556 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:33:21,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:33:21,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00238094 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:33:31,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:33:31,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104668 s, 391 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:33:31,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:33:31,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807367 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14430::WARNING::2013-03-17 17:33:37,687::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:33:37,691::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:33:41,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:33:41,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854468 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:33:41,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:33:41,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00275969 s, 1.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:33:47,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:33:47,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:33:48,153::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:33:48,160::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:33:48,161::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:33:51,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:33:51,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:33:51,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010979 s, 373 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:33:51,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133949 s, 306 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:34:01,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:01,614::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:34:01,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:34:01,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:34:01,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166545 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:01,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00143217 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:01,667::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00143217 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:34:01,667::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:34:01,668::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:34:01,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:34:01,670::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:34:01,671::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:34:01,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:34:01,672::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:34:01,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:34:01,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:02,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:02,116::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:34:02,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:34:02,119::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:34:02,121::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:34:02,122::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:34:02,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:02,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:02,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:02,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022559 s, 182 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:34:11,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:34:11,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145857 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:12,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:12,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129123 s, 317 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:34:21,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:34:21,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763001 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:22,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:22,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735387 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:34:31,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:34:31,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821286 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:32,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:32,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00897423 s, 456 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:34:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:34:41,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127245 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:42,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:42,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135476 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14431::WARNING::2013-03-17 17:34:48,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:34:48,191::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:34:51,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:34:51,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152778 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:34:52,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:34:53,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181425 s, 226 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:34:58,193::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:34:58,194::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:34:58,195::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:34:58,700::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:34:58,707::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:34:58,708::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:35:01,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:35:01,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0095884 s, 427 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:35:03,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:35:03,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191211 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:35:11,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:35:11,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797335 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:35:13,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:35:13,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116314 s, 352 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:35:21,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:35:21,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925237 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:35:23,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:35:23,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845431 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:35:31,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:35:32,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935796 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:35:33,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:35:33,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159116 s, 257 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:35:42,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:35:42,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113279 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:35:43,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:35:43,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792747 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:35:52,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:35:52,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00461999 s, 887 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:35:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:35:53,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146484 s, 280 kB/s\n'; <rc> = 0 >Thread-14432::WARNING::2013-03-17 17:35:58,750::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:35:58,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:36:02,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:36:02,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849282 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:36:03,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:36:03,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0362482 s, 113 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:36:08,755::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:36:08,756::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:36:09,225::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:36:09,231::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:36:09,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:36:12,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:36:12,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017555 s, 233 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:36:13,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:36:13,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158761 s, 258 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:36:22,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:36:22,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240406 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:36:23,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:36:23,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849588 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:36:32,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:36:32,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131589 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:36:33,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:36:33,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195455 s, 210 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:36:42,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:36:42,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078254 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:36:43,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:36:43,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161938 s, 253 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:36:52,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:36:52,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148091 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:36:54,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:36:54,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887274 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:02,385::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:37:02,387::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:37:02,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:02,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128607 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:02,427::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128607 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:37:02,428::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:37:02,428::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:37:02,429::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:37:02,432::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:02,433::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:02,433::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:02,434::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:02,434::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:02,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:02,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:02,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:37:02,879::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:37:02,880::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:37:02,880::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:37:02,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:03,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:03,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:03,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141016 s, 290 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:37:04,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:37:04,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228947 s, 179 kB/s\n'; <rc> = 0 >Thread-14433::WARNING::2013-03-17 17:37:09,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:37:09,263::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:37:13,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:13,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798353 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:37:14,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:37:14,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194003 s, 211 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:37:19,266::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:37:19,268::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:37:19,750::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:37:19,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:37:19,758::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:37:23,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:23,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194171 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:37:24,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:37:24,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104457 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:33,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:33,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841645 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:37:34,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:37:34,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756041 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:43,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:43,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099177 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:37:44,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:37:44,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0097013 s, 422 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:37:53,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:37:53,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148512 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:37:54,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:37:54,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148824 s, 275 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:38:03,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:38:03,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166115 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:38:04,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:38:04,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180215 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:38:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:38:13,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880648 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:38:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:38:14,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113447 s, 361 kB/s\n'; <rc> = 0 >Thread-14434::WARNING::2013-03-17 17:38:19,784::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:38:19,787::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:38:23,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:38:23,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841641 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:38:24,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:38:24,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:38:29,789::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:38:29,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:38:30,248::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:38:30,255::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:38:30,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:38:33,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:38:33,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106079 s, 386 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:38:34,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:38:34,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018065 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:38:43,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:38:43,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161127 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:38:44,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:38:44,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872837 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:38:53,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:38:53,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150713 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:38:55,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:38:55,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959986 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:39:03,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:39:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00678674 s, 604 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:05,084::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:39:05,086::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:39:05,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:05,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015274 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:05,133::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015274 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:39:05,134::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:39:05,135::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:39:05,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:39:05,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:39:05,138::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:39:05,139::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:39:05,139::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:39:05,139::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:39:05,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:05,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:05,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:39:05,584::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:39:05,586::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:39:05,588::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:39:05,588::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:39:05,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:05,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:05,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:05,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111012 s, 369 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:39:13,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:39:13,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932767 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:16,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:16,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151811 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:39:23,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:39:23,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200745 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:26,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:26,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128696 s, 3.2 MB/s\n'; <rc> = 0 >Thread-14435::WARNING::2013-03-17 17:39:30,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:39:30,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:39:33,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:39:33,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196737 s, 208 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:36,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:36,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769367 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:39:40,287::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:39:40,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:39:40,753::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:39:40,759::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:39:40,761::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:39:43,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:39:43,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110248 s, 372 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:46,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:46,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0330331 s, 124 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:39:53,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:39:54,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833751 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:39:56,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:39:56,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821365 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:40:04,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:40:04,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114278 s, 358 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:40:06,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:40:06,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145803 s, 281 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:40:14,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:40:14,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852591 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:40:16,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:40:16,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0057952 s, 707 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:40:24,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:40:24,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152081 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:40:26,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:40:26,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108999 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:40:34,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:40:34,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811808 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:40:36,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:40:36,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120897 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14436::WARNING::2013-03-17 17:40:40,789::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:40:40,794::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:40:44,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:40:44,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013576 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:40:46,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:40:46,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167184 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:40:50,797::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:40:50,798::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:40:50,798::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:40:51,217::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:40:51,224::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:40:51,225::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:40:54,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:40:54,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181315 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:40:56,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:40:56,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00980242 s, 418 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:41:04,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:41:04,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0274938 s, 149 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:41:06,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:41:06,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194193 s, 211 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:41:14,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:41:14,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001667 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:41:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:41:17,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786101 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:41:24,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:41:24,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709555 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:41:27,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:41:27,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869775 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:41:34,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:41:34,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912356 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:41:37,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:41:37,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857613 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:41:44,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:41:44,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912839 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:41:47,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:41:47,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737425 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14437::WARNING::2013-03-17 17:41:51,263::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:41:51,267::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:41:54,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:41:54,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206648 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:41:57,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:41:57,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162926 s, 251 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:42:01,269::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:42:01,270::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:42:01,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:42:01,997::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:42:01,999::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:04,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:42:04,598::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:42:04,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:04,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0166462 s, 123 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:04,646::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0166462 s, 123 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:42:04,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:42:04,646::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:42:04,648::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:42:04,649::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:04,650::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:04,650::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:04,650::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:04,651::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:04,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:05,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:05,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:42:05,059::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:42:05,062::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:42:05,063::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:42:05,063::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:42:05,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:05,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:05,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:05,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118491 s, 346 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:42:07,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:42:07,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0348664 s, 117 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:15,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:15,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136132 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:42:17,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:42:17,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171133 s, 239 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:25,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:25,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209592 s, 195 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:42:27,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:42:27,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881029 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:35,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:35,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140954 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:42:37,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:42:37,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172962 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:45,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:45,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195094 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:42:47,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:42:47,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00838837 s, 488 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:42:55,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:42:55,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078903 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:42:57,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:42:57,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136512 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14438::WARNING::2013-03-17 17:43:02,025::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:43:02,029::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:43:05,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:43:05,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228468 s, 179 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:43:07,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:43:07,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152876 s, 268 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:43:12,031::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:43:12,032::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:43:12,469::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:43:12,475::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:43:12,477::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:43:15,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:43:15,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127471 s, 321 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:43:18,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:43:18,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188314 s, 218 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:43:25,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:43:25,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253606 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:43:28,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:43:28,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105293 s, 389 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:43:35,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:43:35,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776911 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:43:38,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:43:38,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860754 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:43:45,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:43:45,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860592 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:43:48,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:43:48,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123664 s, 331 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:43:55,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:43:56,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814146 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:43:58,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:43:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138081 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:44:06,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:44:06,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00425949 s, 962 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:08,401::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:44:08,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:44:08,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:08,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0112824 s, 182 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:08,457::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0112824 s, 182 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:44:08,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:44:08,458::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:44:08,460::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:44:08,461::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:44:08,461::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:44:08,462::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:44:08,462::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:44:08,464::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:44:08,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:08,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:08,951::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:44:08,952::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:44:08,954::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:44:08,956::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:44:08,957::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:44:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:09,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:09,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:09,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129323 s, 317 kB/s\n'; <rc> = 0 >Thread-14439::WARNING::2013-03-17 17:44:12,502::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:44:12,509::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:44:16,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:44:16,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826474 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:19,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:19,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891303 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:44:22,517::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:44:22,518::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:44:23,012::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:44:23,019::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:44:23,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:44:26,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:44:26,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667227 s, 614 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:29,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:29,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011169 s, 367 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:44:36,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:44:36,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846943 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:39,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:39,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817629 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:44:46,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:44:46,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955941 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:49,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:49,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158903 s, 258 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:44:56,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:44:56,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148774 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:44:59,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:44:59,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117616 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:45:06,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:45:06,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019654 s, 208 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:45:09,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:45:09,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204692 s, 200 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:45:16,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:45:16,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117576 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:45:19,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:45:19,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134095 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14440::WARNING::2013-03-17 17:45:23,046::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:45:23,049::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:45:26,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:45:26,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00994515 s, 412 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:45:29,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:45:30,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.008185 s, 500 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:45:33,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:45:33,053::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:45:33,487::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:45:33,494::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:45:33,495::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:45:36,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:45:36,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145303 s, 282 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:45:40,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:45:40,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00922662 s, 444 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:45:46,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:45:46,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821384 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:45:50,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:45:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120321 s, 340 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:45:56,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:45:56,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172595 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:46:00,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:46:00,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153736 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:46:06,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:46:06,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142855 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:46:10,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:46:10,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00905095 s, 453 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:46:16,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:46:16,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124427 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:46:20,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:46:20,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751775 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:46:26,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:46:26,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793227 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:46:30,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:46:30,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019345 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14441::WARNING::2013-03-17 17:46:33,524::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:46:33,528::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:46:36,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:46:36,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790601 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:46:40,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:46:40,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783246 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:46:43,530::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:46:43,531::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:46:43,531::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:46:43,984::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:46:43,991::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:46:43,992::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:46:46,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:46:46,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194534 s, 211 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:46:50,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:46:50,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184587 s, 222 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:46:56,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:46:56,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224496 s, 182 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:47:00,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:47:00,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193469 s, 212 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:06,845::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:47:06,846::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:47:06,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:06,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0203026 s, 101 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:06,896::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0203026 s, 101 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:47:06,897::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:47:06,897::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:47:06,900::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:47:06,901::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:06,902::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:06,902::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:06,903::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:06,903::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:06,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:07,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:07,365::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:07,366::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:47:07,368::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:47:07,369::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:47:07,370::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:47:07,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:07,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:07,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:07,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131301 s, 312 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:47:10,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:47:10,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00964008 s, 425 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:17,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:17,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132588 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:47:20,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:47:20,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020172 s, 203 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:27,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:27,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162601 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:47:31,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:47:31,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166492 s, 246 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:47:37,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:37,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157579 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:47:41,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:47:41,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008396 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14442::WARNING::2013-03-17 17:47:44,022::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:47:44,024::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:47:47,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:47,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876019 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:47:51,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:47:51,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0327629 s, 125 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:47:54,032::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:47:54,033::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:47:54,513::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:47:54,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:47:54,521::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:47:58,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:47:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143988 s, 284 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:48:01,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:48:01,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138617 s, 295 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:48:08,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:48:08,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181087 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:48:11,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:48:11,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127987 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:48:18,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:48:18,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773334 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:48:21,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:48:21,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811237 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:48:28,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:48:28,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176817 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:48:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:48:31,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154765 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:48:38,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:48:38,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132581 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:48:41,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:48:41,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017105 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:48:48,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:48:48,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138876 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:48:51,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:48:51,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979785 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14443::WARNING::2013-03-17 17:48:54,547::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:48:54,551::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:48:58,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:48:58,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968376 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:01,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:01,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893223 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:49:04,553::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:49:04,554::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:49:05,026::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:49:05,033::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:49:05,034::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:49:08,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:49:08,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205367 s, 199 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:11,918::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:49:11,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:49:11,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:11,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.014872 s, 138 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:11,964::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.014872 s, 138 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:49:11,965::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:49:11,965::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:49:11,966::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:49:11,968::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:49:11,968::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:49:11,969::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:49:11,969::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:49:11,969::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:49:11,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:12,396::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:49:12,397::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:49:12,401::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:49:12,402::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:49:12,402::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:49:12,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:12,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:12,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:12,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107552 s, 381 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:49:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:49:18,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137188 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:22,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:22,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077805 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:49:28,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:49:28,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897536 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:32,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:33,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826892 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:49:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:49:38,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861571 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:43,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:43,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148106 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:49:48,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:49:48,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141971 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:49:53,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:49:53,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806466 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:49:58,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:49:58,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082221 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:50:03,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:50:03,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121803 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14444::WARNING::2013-03-17 17:50:05,060::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:50:05,063::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:50:08,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:50:08,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001047 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:50:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:50:13,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158216 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:50:15,066::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:50:15,067::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:50:15,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:50:15,489::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:50:15,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:50:18,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:50:18,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018474 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:50:23,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:50:23,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019936 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:50:28,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:50:28,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120159 s, 341 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:50:33,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:50:33,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127195 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:50:38,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:50:38,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885765 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:50:43,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:50:43,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858199 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:50:48,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:50:48,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970615 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:50:53,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:50:53,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143775 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:50:58,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:50:58,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00457244 s, 896 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:51:03,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:51:03,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152346 s, 269 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:51:08,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:51:08,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951148 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:51:13,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:51:13,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110022 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14445::WARNING::2013-03-17 17:51:15,519::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:51:15,526::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:51:18,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:51:18,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885716 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:51:23,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:51:23,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785213 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:51:25,529::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:51:25,530::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:51:25,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:51:25,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:51:25,989::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:51:28,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:51:28,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972676 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:51:33,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:51:34,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00963113 s, 425 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:51:38,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:51:39,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206558 s, 198 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:51:44,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:51:44,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187135 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:51:49,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:51:49,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020809 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:51:54,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:51:54,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134717 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:51:59,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:51:59,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850359 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:52:04,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:52:04,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141922 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:52:09,131::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:52:09,132::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:52:09,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:09,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0221935 s, 92.3 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:52:09,190::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0221935 s, 92.3 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:52:09,190::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:52:09,191::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:52:09,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:52:09,193::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:09,194::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:09,195::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:09,195::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:09,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:09,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:09,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:52:09,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:09,633::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:52:09,636::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:52:09,637::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:52:09,637::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:52:09,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:10,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:52:10,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:10,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00895415 s, 457 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:52:14,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:52:14,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184173 s, 222 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:52:20,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:20,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216709 s, 189 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:52:24,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:52:24,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111661 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14446::WARNING::2013-03-17 17:52:26,020::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:52:26,026::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:52:30,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:30,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708742 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:52:34,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:52:34,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091325 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:52:36,029::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:52:36,030::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:52:36,031::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:52:36,514::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:52:36,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:52:36,522::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:52:40,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:40,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117832 s, 348 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:52:44,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:52:44,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144272 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:52:50,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:52:50,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868842 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:52:54,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:52:54,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151658 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:53:00,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:53:00,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715769 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:53:04,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:53:04,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133323 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:53:10,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:53:10,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171864 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:53:14,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:53:14,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880196 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:53:20,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:53:20,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02032 s, 202 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:53:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:53:24,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799761 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:53:30,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:53:30,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851788 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:53:35,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:53:35,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116058 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14447::WARNING::2013-03-17 17:53:36,554::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:53:36,556::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:53:40,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:53:40,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937466 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:53:45,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:53:45,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174734 s, 234 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:53:46,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:53:46,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:53:47,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:53:47,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:53:47,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:53:50,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:53:50,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00984308 s, 416 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:53:55,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:53:55,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153738 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:54:00,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:54:00,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216717 s, 189 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:05,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:05,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713959 s, 574 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:54:10,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:54:10,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157259 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:15,360::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:54:15,362::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:54:15,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:15,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000690629 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:15,391::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000690629 s, 3.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:54:15,391::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:54:15,391::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:54:15,393::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:54:15,394::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:54:15,395::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:54:15,397::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:54:15,398::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:54:15,398::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:54:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:15,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:15,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:54:15,833::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:54:15,836::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:54:15,837::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:54:15,838::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:54:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:16,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:16,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:16,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00533166 s, 768 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:54:20,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:54:20,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00682336 s, 600 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:26,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:26,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00839643 s, 488 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:54:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:54:30,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217009 s, 189 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:36,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:36,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825981 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:54:40,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:54:40,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011374 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:46,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:46,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103504 s, 396 kB/s\n'; <rc> = 0 >Thread-14448::WARNING::2013-03-17 17:54:47,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:54:47,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:54:50,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:54:50,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176573 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:54:56,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:54:56,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727758 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:54:57,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:54:57,116::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:54:57,579::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:54:57,586::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:54:57,587::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:55:00,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:55:00,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181925 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:55:07,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:55:07,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177746 s, 230 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:55:10,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:55:10,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947241 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:55:17,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:55:17,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816082 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:55:20,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:55:20,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767902 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:55:27,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:55:27,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02235 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:55:30,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:55:30,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151802 s, 270 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:55:37,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:55:37,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152892 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:55:41,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:55:41,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152476 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:55:47,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:55:47,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959886 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:55:51,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:55:51,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098102 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:55:57,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:55:57,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773261 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14449::WARNING::2013-03-17 17:55:57,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:55:57,624::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:56:01,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:56:01,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778221 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:56:07,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:56:07,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953461 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:56:07,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:56:07,753::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:56:08,259::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:56:08,267::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:56:08,282::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:56:11,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:56:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103133 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:56:17,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:56:17,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769133 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:56:21,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:56:21,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213224 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:56:27,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:56:27,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133974 s, 306 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:56:31,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:56:31,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154742 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:56:37,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:56:38,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880344 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:56:41,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:56:41,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808728 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:56:48,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:56:48,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147243 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:56:51,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:56:51,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161787 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:56:58,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:56:58,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153139 s, 267 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:01,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:01,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954158 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:57:08,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:08,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757182 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14450::WARNING::2013-03-17 17:57:08,337::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:57:08,340::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:57:11,441::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 17:57:11,442::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:57:11,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:11,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118961 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:11,478::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118961 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 17:57:11,478::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 17:57:11,479::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 17:57:11,480::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 17:57:11,481::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:11,482::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:11,482::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:11,482::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:11,483::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:11,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:11,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:11,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:11,883::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 17:57:11,885::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 17:57:11,886::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 17:57:11,887::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 17:57:11,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:12,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:12,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:12,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0248285 s, 165 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:57:18,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:57:18,343::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:18,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:18,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159952 s, 256 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:57:18,975::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:57:18,981::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:57:18,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:57:22,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:22,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812501 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:57:28,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:28,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753265 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:32,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:32,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800733 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:57:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:38,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141531 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:42,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:42,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755046 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:57:48,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:48,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917106 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:57:52,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:57:52,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866498 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:57:58,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:57:58,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710965 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:58:02,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:58:02,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741463 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:58:08,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:08,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863147 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:58:12,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:58:12,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728511 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:58:18,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:18,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168792 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14451::WARNING::2013-03-17 17:58:19,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:58:19,026::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 17:58:22,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:58:22,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088154 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:58:29,040::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 17:58:29,042::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:58:29,043::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:29,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:29,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926406 s, 442 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:58:29,574::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:58:29,579::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:58:29,581::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 17:58:32,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:58:32,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013489 s, 304 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:58:39,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:39,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975275 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:58:42,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:58:42,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205182 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:58:49,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:49,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749151 s, 547 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:58:52,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:58:52,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754212 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:58:59,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:58:59,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760194 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:59:02,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:59:02,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187158 s, 219 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:59:09,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:09,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175452 s, 233 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:59:12,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:59:12,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779497 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:59:19,578::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 17:59:19,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:59:19,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:19,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149382 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:59:19,620::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149382 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 17:59:19,620::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 17:59:19,620::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 17:59:19,623::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 17:59:19,624::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:19,624::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:19,625::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:19,625::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:19,625::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:19,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:20,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:59:20,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:20,134::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 17:59:20,137::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 17:59:20,138::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 17:59:20,138::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 17:59:20,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:20,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:59:20,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:20,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190252 s, 215 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:59:22,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:59:22,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107375 s, 381 kB/s\n'; <rc> = 0 >Thread-14452::WARNING::2013-03-17 17:59:29,624::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 17:59:29,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 17:59:30,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:30,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826725 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:59:32,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:59:32,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867719 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 17:59:39,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 17:59:39,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 17:59:40,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 17:59:40,075::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 17:59:40,076::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 17:59:40,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:40,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143964 s, 285 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:59:42,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:59:42,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181462 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 17:59:50,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 17:59:50,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802633 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 17:59:52,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 17:59:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156854 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:00:00,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:00:00,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176773 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:00:03,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:00:03,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833354 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:00:10,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:00:11,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117755 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:00:13,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:00:13,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143702 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:00:21,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:00:21,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975287 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:00:23,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:00:23,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078057 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:00:31,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:00:31,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142875 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:00:33,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:00:33,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126079 s, 325 kB/s\n'; <rc> = 0 >Thread-14453::WARNING::2013-03-17 18:00:40,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:00:40,110::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:00:41,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:00:41,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097075 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:00:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:00:43,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079596 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:00:50,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:00:50,122::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:00:50,619::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:00:50,628::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:00:50,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:00:51,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:00:51,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638406 s, 642 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:00:53,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:00:53,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181668 s, 225 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:01:01,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:01:01,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704335 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:01:03,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:01:03,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849206 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:01:11,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:01:11,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129203 s, 317 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:01:13,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:01:13,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708737 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:01:21,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:01:21,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844209 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:01:23,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:01:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140806 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:01:31,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:01:31,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167297 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:01:33,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:01:33,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142303 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:01:41,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:01:41,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103635 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:01:43,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:01:43,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225447 s, 182 kB/s\n'; <rc> = 0 >Thread-14454::WARNING::2013-03-17 18:01:50,659::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:01:50,664::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:01:51,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:01:51,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874709 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:01:53,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:01:53,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896306 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:02:00,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:02:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:02:01,184::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:02:01,192::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:02:01,198::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:02:01,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:02:01,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224398 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:03,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:03,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179808 s, 228 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:02:11,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:02:11,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139481 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:13,638::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:02:13,640::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:02:13,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:13,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00127336 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:13,681::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00127336 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:02:13,682::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:02:13,682::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:02:13,684::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:02:13,685::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:02:13,686::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:02:13,686::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:02:13,687::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:02:13,687::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:02:13,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:14,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:14,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:02:14,133::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:02:14,135::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:02:14,136::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:02:14,137::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:02:14,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:14,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:14,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:14,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163754 s, 250 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:02:22,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:02:22,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00827637 s, 495 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:24,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:24,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919515 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:02:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:02:32,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744755 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:34,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:34,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152485 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:02:42,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:02:42,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101581 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:44,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:44,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820878 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:02:52,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:02:52,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144334 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:02:54,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:02:54,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912209 s, 4.5 MB/s\n'; <rc> = 0 >Thread-14455::WARNING::2013-03-17 18:03:01,230::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:03:01,232::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:03:02,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:03:02,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180325 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:03:04,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:03:04,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829446 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:03:11,235::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:03:11,236::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:03:11,787::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:03:11,793::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:03:11,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:03:12,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:03:12,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174228 s, 235 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:03:14,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:03:14,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188731 s, 217 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:03:22,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:03:22,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866416 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:03:24,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:03:24,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224622 s, 182 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:03:32,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:03:32,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141561 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:03:34,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:03:34,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220743 s, 186 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:03:42,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:03:42,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013697 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:03:44,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:03:44,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142621 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:03:52,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:03:52,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145913 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:03:54,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:03:55,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146823 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:02,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:02,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753675 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:04:05,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:04:05,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175584 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14456::WARNING::2013-03-17 18:04:11,819::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:04:11,823::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:04:12,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:12,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00368524 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:04:15,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:04:15,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703162 s, 5.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:04:21,831::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:04:21,832::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:04:21,833::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:04:22,307::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:04:22,314::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:04:22,315::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,000::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:04:23,004::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:04:23,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:23,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0130118 s, 157 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:23,043::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0130118 s, 157 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:04:23,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:04:23,044::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:04:23,046::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:04:23,047::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,047::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,048::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,048::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,049::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:23,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:23,446::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:04:23,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:04:23,449::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:04:23,450::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:04:23,451::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:04:23,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:23,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:23,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:23,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113718 s, 360 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:04:25,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:04:25,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140807 s, 291 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:33,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:33,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00683755 s, 599 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:04:35,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:04:35,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883446 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:43,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:43,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222758 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:04:45,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:04:45,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138399 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:04:54,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:04:54,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774404 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:04:55,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:04:55,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014335 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:05:04,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:05:04,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826164 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:05:05,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:05:05,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022967 s, 178 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:05:14,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:05:14,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014821 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:05:15,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:05:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149865 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14457::WARNING::2013-03-17 18:05:22,342::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:05:22,345::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:05:24,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:05:24,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166519 s, 246 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:05:25,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:05:25,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817281 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:05:32,347::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:05:32,349::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:05:32,831::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:05:32,838::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:05:32,839::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:05:34,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:05:34,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00775243 s, 528 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:05:35,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:05:35,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0277424 s, 148 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:05:44,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:05:44,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162591 s, 252 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:05:45,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:05:45,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825565 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:05:54,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:05:54,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763662 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:05:55,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:05:55,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154957 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:06:04,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:06:04,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162207 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:06:05,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:06:05,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145337 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:06:14,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:06:14,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125538 s, 326 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:06:15,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:06:15,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00330008 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:06:24,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:06:24,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140978 s, 291 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:06:25,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:06:25,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147726 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14458::WARNING::2013-03-17 18:06:32,866::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:06:32,870::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:06:34,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:06:34,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153112 s, 268 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:06:35,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:06:35,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077095 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:06:42,872::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:06:42,873::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:06:43,308::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:06:43,314::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:06:43,315::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:06:44,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:06:44,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103901 s, 394 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:06:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:06:45,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170806 s, 240 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:06:55,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:06:55,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787785 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:06:55,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:06:55,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843035 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:07:05,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:07:05,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151261 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:05,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:05,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832821 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:07:15,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:07:15,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784035 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:15,887::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:07:15,889::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:07:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:15,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000861269 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:15,924::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000861269 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:07:15,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:07:15,925::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:07:15,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:07:15,929::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:07:15,930::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:07:15,930::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:07:15,931::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:07:15,932::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:07:15,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:16,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:16,388::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:07:16,389::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:07:16,393::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:07:16,394::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:07:16,395::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:07:16,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:16,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:16,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:16,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00925693 s, 442 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:07:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:07:25,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136817 s, 299 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:26,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:26,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220427 s, 186 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:07:35,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:07:35,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142744 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:36,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:36,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196595 s, 208 kB/s\n'; <rc> = 0 >Thread-14459::WARNING::2013-03-17 18:07:43,340::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:07:43,343::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:07:45,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:07:45,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00675866 s, 606 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:46,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:46,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093155 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:07:53,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:07:53,375::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:07:53,882::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:07:53,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:07:53,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:07:55,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:07:55,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00917471 s, 446 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:07:56,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:07:57,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117935 s, 347 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:08:05,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:08:05,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109741 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:08:07,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:08:07,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256479 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:08:15,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:08:15,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772316 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:08:17,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:08:17,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001441 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:08:25,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:08:25,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744107 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:08:27,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:08:27,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102582 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:08:35,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:08:35,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859516 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:08:37,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:08:37,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016547 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:08:45,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:08:45,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839323 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:08:47,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:08:47,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932022 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14460::WARNING::2013-03-17 18:08:53,927::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:08:53,931::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:08:55,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:08:55,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790258 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:08:57,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:08:57,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988977 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:09:03,933::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:09:03,934::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:09:04,425::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:09:04,431::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:09:04,433::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:06,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:06,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118868 s, 345 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:09:07,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:09:07,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119791 s, 342 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:16,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:16,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144351 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:09:17,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:09:17,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998397 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:26,182::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:09:26,183::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:09:26,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:26,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000895121 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:26,209::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000895121 s, 2.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:09:26,210::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:09:26,210::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:09:26,212::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:09:26,213::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:26,214::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:26,214::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:26,215::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:26,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:26,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:26,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:26,649::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:09:26,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:09:26,652::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:09:26,653::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:09:26,654::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:09:26,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:27,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:27,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:27,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00962118 s, 426 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:09:27,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:09:27,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162342 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:37,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:37,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124237 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:09:37,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:09:37,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146691 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:47,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:47,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166569 s, 246 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:09:47,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:09:47,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803811 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:09:57,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:09:57,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810884 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:09:57,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:09:57,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135541 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14461::WARNING::2013-03-17 18:10:04,458::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:10:04,462::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:10:07,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:10:07,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170192 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:10:07,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:10:07,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790396 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:10:14,467::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:10:14,468::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:10:14,469::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:10:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:10:14,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:10:14,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:10:17,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:10:17,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00628958 s, 651 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:10:17,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:10:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204879 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:10:27,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:10:27,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825461 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:10:27,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:10:27,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995502 s, 411 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:10:37,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:10:37,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078922 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:10:37,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:10:37,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00440291 s, 930 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:10:47,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:10:47,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015907 s, 257 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:10:47,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:10:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926636 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:10:57,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:10:57,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135945 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:10:57,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:10:57,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720227 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:11:07,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:07,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0312964 s, 131 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:11:07,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:11:07,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104442 s, 3.9 MB/s\n'; <rc> = 0 >Thread-14462::WARNING::2013-03-17 18:11:14,929::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:11:14,932::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:11:17,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:18,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:11:18,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00374058 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:11:18,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125703 s, 326 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:11:24,936::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:11:24,937::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:11:25,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:11:25,338::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:11:25,340::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:11:28,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:28,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:11:28,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871452 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:11:28,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182162 s, 225 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:11:38,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:38,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:11:38,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108841 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:11:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961318 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:11:48,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:48,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:11:48,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00381288 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:11:48,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900653 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:11:58,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:58,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:11:58,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114806 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:11:58,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168474 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:08,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:12:08,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:12:08,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117356 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:08,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199521 s, 205 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:18,749::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:12:18,750::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:12:18,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:18,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00150418 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:18,794::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00150418 s, 1.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:12:18,794::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:12:18,794::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:12:18,797::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:12:18,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:18,835::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:12:18,858::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:12:18,864::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:12:18,866::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:12:18,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112594 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:18,879::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:12:18,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:19,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:19,327::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:12:19,328::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:12:19,331::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:12:19,332::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:12:19,332::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:12:19,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:19,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:19,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:19,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0496526 s, 82.5 kB/s\n'; <rc> = 0 >Thread-14463::WARNING::2013-03-17 18:12:25,365::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:12:25,370::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:12:28,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:12:29,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00907251 s, 451 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:29,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:29,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881054 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:12:35,372::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:12:35,373::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:12:35,868::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:12:35,875::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:12:35,877::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:12:39,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:12:39,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118976 s, 344 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:39,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:39,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145325 s, 282 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:12:49,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:12:49,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782409 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:49,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:12:49,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735203 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:12:59,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:12:59,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150639 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:12:59,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:13:00,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860312 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:13:09,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:13:09,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136332 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:13:10,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:13:10,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014212 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:13:19,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:13:19,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860292 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:13:20,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:13:20,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151183 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:13:29,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:13:29,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00997618 s, 411 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:13:30,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:13:30,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022924 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14464::WARNING::2013-03-17 18:13:35,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:13:35,904::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:13:39,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:13:39,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010245 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:13:40,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:13:40,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799439 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:13:45,907::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:13:45,909::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:13:46,313::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:13:46,320::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:13:46,321::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:13:49,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:13:49,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00649246 s, 631 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:13:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:13:50,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114245 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:13:59,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:13:59,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750673 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:14:00,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:00,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126945 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:09,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:09,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154986 s, 264 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:14:10,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:10,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309019 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:19,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:19,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103012 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:14:20,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:20,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161036 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:29,880::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:14:29,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:14:29,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:29,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0101054 s, 203 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:29,925::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0101054 s, 203 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:14:29,926::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:14:29,926::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:14:29,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:14:29,928::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:14:29,929::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:14:29,930::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:14:29,930::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:14:29,930::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:14:29,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:30,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:30,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:30,418::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:14:30,419::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:14:30,421::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:14:30,423::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:14:30,423::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:14:30,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:30,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203193 s, 202 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:30,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:30,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:30,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127156 s, 322 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:14:40,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173285 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:40,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:40,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763054 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14465::WARNING::2013-03-17 18:14:46,349::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:14:46,357::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:14:50,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:14:50,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158803 s, 258 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:14:50,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:14:51,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158868 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:14:56,369::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:14:56,370::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:14:56,852::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:14:56,858::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:14:56,860::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:15:00,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:15:00,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186113 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:15:01,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:15:01,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882138 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:15:10,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:15:10,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147724 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:15:11,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:15:11,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127247 s, 322 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:15:20,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:15:20,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137288 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:15:21,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:15:21,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151126 s, 271 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:15:30,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:15:30,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927393 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:15:31,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:15:31,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141327 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:15:40,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:15:40,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569254 s, 720 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:15:41,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:15:41,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134648 s, 304 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:15:50,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:15:50,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140934 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:15:51,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:15:51,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667732 s, 613 kB/s\n'; <rc> = 0 >Thread-14466::WARNING::2013-03-17 18:15:56,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:15:56,904::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:16:00,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:16:01,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145793 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:16:01,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:16:01,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107386 s, 381 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:16:06,905::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:16:06,907::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:16:06,908::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:16:07,322::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:16:07,328::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:16:07,331::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:16:11,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:16:11,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110608 s, 370 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:16:11,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:16:11,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00668177 s, 613 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:16:21,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:16:21,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165233 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:16:21,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:16:21,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398174 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:16:31,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:16:31,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113946 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:16:31,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:16:31,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140249 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:16:41,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:16:41,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022778 s, 180 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:16:41,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:16:41,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148504 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:16:51,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:16:51,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154787 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:16:52,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:16:52,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799943 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:01,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:01,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0249338 s, 164 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:17:02,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:17:02,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106245 s, 3.9 MB/s\n'; <rc> = 0 >Thread-14467::WARNING::2013-03-17 18:17:07,364::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:17:07,370::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:17:11,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:11,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825314 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:17:12,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:17:12,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789477 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:17:17,372::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:17:17,373::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:17:17,768::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:17:17,775::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:17:17,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,469::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:17:21,472::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:17:21,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:21,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000811196 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:21,500::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000811196 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:17:21,501::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:17:21,501::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:17:21,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:17:21,504::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,504::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,505::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,505::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,505::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:21,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:21,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:17:21,883::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:17:21,885::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:17:21,886::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:17:21,887::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:17:21,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:22,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:22,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:17:22,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135832 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:17:22,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104384 s, 392 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:32,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:32,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828632 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:17:32,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:17:32,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814514 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:42,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079102 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:17:42,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:17:42,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859722 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:17:52,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:17:52,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967412 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:17:52,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:17:52,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793429 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:18:02,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:18:02,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148121 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:18:02,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:18:02,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00698631 s, 586 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:18:12,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:18:12,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212782 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:18:12,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:18:12,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278021 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14468::WARNING::2013-03-17 18:18:17,803::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:18:17,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:18:22,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:18:22,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884818 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:18:22,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:18:22,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743398 s, 551 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:18:27,808::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:18:27,809::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:18:28,299::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:18:28,307::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:18:28,309::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:18:32,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:18:32,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075665 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:18:32,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:18:32,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201257 s, 204 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:18:42,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:18:42,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146376 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:18:42,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:18:42,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00712009 s, 575 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:18:52,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:18:52,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823783 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:18:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:18:53,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014892 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:19:02,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:19:02,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084242 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:03,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:03,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111202 s, 368 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:19:12,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:19:12,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143109 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:13,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:13,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230957 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:19:22,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:19:22,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129633 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:23,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:23,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105737 s, 3.9 MB/s\n'; <rc> = 0 >Thread-14469::WARNING::2013-03-17 18:19:28,334::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:19:28,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:19:32,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:19:32,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857448 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:33,419::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:19:33,421::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:19:33,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:33,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00169421 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:33,497::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00169421 s, 1.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:19:33,497::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:19:33,497::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:19:33,499::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:19:33,500::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:19:33,501::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:19:33,501::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:19:33,502::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:19:33,502::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:19:33,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:33,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:33,931::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:19:33,932::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:19:33,935::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:19:33,936::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:19:33,936::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:19:33,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:34,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:34,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:34,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173039 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:19:38,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:19:38,342::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:19:38,765::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:19:38,771::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:19:38,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:19:42,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:19:42,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114311 s, 358 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:44,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:44,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821942 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:19:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:19:52,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220715 s, 186 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:19:54,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:19:54,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145513 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:20:02,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:20:03,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209198 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:20:04,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:20:04,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787992 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:20:13,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:20:13,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814094 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:20:14,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:20:14,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014464 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:20:23,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:20:23,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221074 s, 185 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:20:24,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:20:24,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854378 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:20:33,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:20:33,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000714729 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:20:34,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:20:34,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794662 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14470::WARNING::2013-03-17 18:20:38,804::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:20:38,807::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:20:43,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:20:43,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756831 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:20:44,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:20:44,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279714 s, 1.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:20:48,810::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:20:48,811::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:20:49,278::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:20:49,285::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:20:49,286::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:20:53,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:20:53,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173802 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:20:54,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:20:55,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148711 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:21:03,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:21:03,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799693 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:21:05,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:21:05,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937297 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:21:13,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:21:13,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855962 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:21:15,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:21:15,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788209 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:21:23,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:21:23,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919328 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:21:25,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:21:25,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015178 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:21:33,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:21:33,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00324404 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:21:35,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:21:35,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097349 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:21:43,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:21:43,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779326 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:21:45,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:21:45,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145129 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14471::WARNING::2013-03-17 18:21:49,319::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:21:49,324::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:21:53,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:21:53,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00390587 s, 1.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:21:55,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:21:55,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847015 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:21:59,327::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:21:59,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:21:59,329::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:21:59,745::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:21:59,752::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:21:59,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:03,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:03,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119824 s, 342 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:22:05,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:22:05,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149932 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:13,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:13,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140543 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:22:15,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:22:15,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019494 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:23,598::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:22:23,599::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:22:23,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:23,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000747612 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:23,631::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000747612 s, 2.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:22:23,632::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:22:23,632::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:22:23,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:22:23,635::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:23,636::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:23,636::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:23,636::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:23,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:23,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:24,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:24,051::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:22:24,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:22:24,055::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:22:24,058::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:22:24,058::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:22:24,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:24,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:24,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:24,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586101 s, 699 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:22:25,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:22:25,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810584 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:34,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:34,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799342 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:22:35,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:22:35,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145856 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:44,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:44,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839004 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:22:45,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:22:45,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151516 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:22:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:22:54,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101637 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:22:55,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:22:56,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167511 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14472::WARNING::2013-03-17 18:22:59,791::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:22:59,794::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:23:04,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:23:04,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829941 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:23:06,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:23:06,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825824 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:23:09,799::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:23:09,800::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:23:10,223::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:23:10,230::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:23:10,231::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:23:14,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:23:14,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857679 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:23:16,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:23:16,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840426 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:23:24,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:23:24,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888133 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:23:26,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:23:26,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802199 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:23:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:23:34,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845356 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:23:36,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:23:36,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137458 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:23:44,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:23:44,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743244 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:23:46,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:23:46,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757012 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:23:54,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:23:54,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128925 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:23:56,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:23:56,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967045 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:24:04,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:24:04,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130498 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:06,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:06,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232765 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14473::WARNING::2013-03-17 18:24:10,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:24:10,265::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:24:14,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:24:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023528 s, 174 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:16,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:16,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783059 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:24:20,269::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:24:20,270::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:24:20,701::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:24:20,708::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:24:20,709::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:24:24,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:24:24,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901118 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:26,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:26,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130694 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:24:34,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:24:34,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083147 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:36,757::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:24:36,759::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:24:36,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:36,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000756357 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:36,796::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000756357 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:24:36,796::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:24:36,797::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:24:36,798::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:24:36,800::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:24:36,801::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:24:36,801::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:24:36,801::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:24:36,802::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:24:36,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:37,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:37,220::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:24:37,221::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:24:37,223::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:24:37,225::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:24:37,225::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:24:37,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:37,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:37,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:37,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121073 s, 338 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:24:44,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:24:45,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882116 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:47,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:47,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159656 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:24:55,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:24:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156041 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:24:57,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:24:57,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824586 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:25:05,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:25:05,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146248 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:25:07,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:25:07,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138258 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:25:15,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:25:15,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127971 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:25:17,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:25:17,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00199261 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14474::WARNING::2013-03-17 18:25:20,737::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:25:20,740::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:25:25,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:25:25,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925489 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:25:27,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:25:28,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789244 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:25:30,751::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:25:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:25:31,232::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:25:31,239::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:25:31,240::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:25:35,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:25:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184165 s, 222 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:25:38,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:25:38,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865952 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:25:45,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:25:45,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758993 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:25:48,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:25:48,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762454 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:25:55,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:25:55,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102566 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:25:58,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:25:58,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990605 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:26:05,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:26:05,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149896 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:26:08,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:26:08,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828865 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:26:15,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:26:15,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165261 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:26:18,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:26:18,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144105 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:26:25,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:26:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152017 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:26:28,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:26:28,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774218 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14475::WARNING::2013-03-17 18:26:31,266::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:26:31,270::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:26:35,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:26:35,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081919 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:26:38,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:26:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104011 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:26:41,272::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:26:41,273::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:26:41,730::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:26:41,738::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:26:41,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:26:45,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:26:45,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831519 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:26:48,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:26:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127207 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:26:55,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:26:55,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140118 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:26:58,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:26:58,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837423 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:05,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:05,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161594 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:27:08,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:27:08,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142126 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:15,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:15,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229759 s, 178 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:27:18,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:27:18,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210296 s, 195 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:25,710::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:27:25,711::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:27:25,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:25,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000826097 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:25,740::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000826097 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:27:25,741::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:27:25,741::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:27:25,743::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:27:25,744::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:25,745::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:25,745::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:25,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:25,746::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:25,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:26,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:26,178::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:26,179::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:27:26,181::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:27:26,182::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:27:26,183::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:27:26,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:26,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:26,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00323605 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:27:28,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:27:28,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172211 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:27:36,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:36,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111598 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:27:39,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:27:39,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236553 s, 173 kB/s\n'; <rc> = 0 >Thread-14476::WARNING::2013-03-17 18:27:41,774::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:27:41,777::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:27:46,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:46,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229257 s, 179 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:27:49,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:27:49,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804408 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:27:51,778::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:27:51,780::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:27:51,780::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:27:52,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:27:52,210::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:27:52,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:27:56,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:27:56,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232347 s, 176 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:27:59,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:27:59,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847858 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:28:06,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:28:06,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143133 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:28:09,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:28:09,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142395 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:28:16,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:28:16,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144025 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:28:19,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:28:19,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753335 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:28:26,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:28:26,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997065 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:28:29,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:28:29,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958577 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:28:36,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:28:36,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752502 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:28:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:28:39,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162889 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:28:46,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:28:46,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162649 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:28:49,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:28:49,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182025 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14477::WARNING::2013-03-17 18:28:52,242::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:28:52,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:28:56,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:28:56,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008077 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:28:59,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:28:59,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818547 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:29:02,248::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:29:02,249::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:29:02,709::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:29:02,716::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:29:02,718::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:29:06,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:29:06,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853157 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:09,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894499 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:29:17,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:29:17,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143538 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:19,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:19,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856654 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:29:27,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:29:27,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212471 s, 193 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:29,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:29,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947815 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:29:37,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:29:37,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014584 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:40,015::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:29:40,016::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:29:40,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:40,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000808071 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:40,047::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000808071 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:29:40,047::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:29:40,047::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:29:40,049::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:29:40,050::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:29:40,051::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:29:40,051::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:29:40,052::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:29:40,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:29:40,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:40,479::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:29:40,480::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:29:40,483::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:29:40,484::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:29:40,484::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:29:40,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:40,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:40,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:40,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00996681 s, 411 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:29:47,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:29:47,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161057 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:29:50,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:29:50,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149742 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:29:57,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:29:57,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941188 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:30:01,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:30:01,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114535 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14478::WARNING::2013-03-17 18:30:02,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:30:02,752::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:30:07,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:30:07,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128964 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:30:11,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:30:11,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144454 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:30:12,819::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:30:12,821::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:30:13,299::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:30:13,306::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:30:13,308::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:30:17,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:30:17,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269339 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:30:21,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:30:21,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840417 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:30:27,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:30:27,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136735 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:30:31,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:30:31,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821391 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:30:37,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:30:37,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785476 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:30:41,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:30:41,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143111 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:30:47,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:30:47,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189161 s, 217 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:30:51,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:30:51,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0305465 s, 134 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:30:57,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:30:57,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111156 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:31:01,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:31:01,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824583 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:31:07,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:31:07,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827714 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:31:11,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:31:11,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100954 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14479::WARNING::2013-03-17 18:31:13,334::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:31:13,341::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:31:17,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:31:17,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795606 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:31:21,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:31:21,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933134 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:31:23,344::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:31:23,345::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:31:23,954::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:31:24,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:31:24,068::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:31:27,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:31:27,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213948 s, 191 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:31:31,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:31:31,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848184 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:31:37,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:31:37,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194599 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:31:41,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:31:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842796 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:31:47,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:31:47,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715446 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:31:51,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:31:52,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008227 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:31:57,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:31:57,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101974 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:32:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:32:02,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165195 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:07,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:07,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850506 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:32:12,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:32:12,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011123 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:17,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:17,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118938 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:32:22,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:32:22,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017326 s, 236 kB/s\n'; <rc> = 0 >Thread-14480::WARNING::2013-03-17 18:32:24,123::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:32:24,128::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:32:27,842::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:32:27,844::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:32:27,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:27,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00106231 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:27,884::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00106231 s, 1.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:32:27,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:32:27,885::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:32:27,886::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:32:27,887::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:27,888::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:27,888::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:27,889::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:27,889::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:27,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:28,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:28,399::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:28,399::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:32:28,402::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:32:28,403::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:32:28,403::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:32:28,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:28,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:28,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:28,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00625484 s, 655 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:32:32,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:32:32,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717012 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:32:34,130::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:32:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:32:34,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:32:34,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:32:34,555::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:32:38,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200258 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:32:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:32:42,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101747 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:48,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:48,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137569 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:32:52,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:32:52,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852569 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:32:58,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:32:58,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122443 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:33:02,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:33:02,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706443 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:33:08,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:33:08,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925271 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:33:12,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:33:12,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151478 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:33:18,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:33:19,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769438 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:33:22,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:33:22,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149736 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:33:29,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:33:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150892 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:33:33,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:33:33,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751045 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14481::WARNING::2013-03-17 18:33:34,589::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:33:34,592::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:33:39,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:33:39,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806718 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:33:43,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:33:43,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896385 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:33:44,597::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:33:44,599::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:33:44,600::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:33:45,009::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:33:45,016::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:33:45,018::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:33:49,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:33:49,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787908 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:33:53,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:33:53,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786611 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:33:59,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:33:59,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193311 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:03,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180476 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:34:09,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:34:09,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013751 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:13,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:13,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780203 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:34:19,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:34:19,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796257 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:23,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164155 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:34:29,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:34:29,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214342 s, 191 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:33,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:33,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859557 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:34:39,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:34:39,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281933 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:43,839::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:34:43,847::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:34:43,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:43,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0178014 s, 115 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:43,953::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0178014 s, 115 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:34:43,954::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:34:43,954::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:34:43,956::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:34:43,958::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:34:43,959::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:34:43,959::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:34:43,960::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:34:43,962::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:34:43,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:44,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:44,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:34:44,594::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:34:44,608::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:34:44,611::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:34:44,612::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:34:44,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:44,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:34:45,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:45,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00216448 s, 1.9 MB/s\n'; <rc> = 0 >Thread-14482::WARNING::2013-03-17 18:34:45,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:34:45,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:34:49,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:34:49,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833484 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:34:55,105::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:34:55,107::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:55,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:34:55,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141227 s, 2.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:34:55,576::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:34:55,582::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:34:55,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:34:59,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:34:59,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127571 s, 321 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:35:05,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:35:05,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149946 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:35:09,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:35:09,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810164 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:35:15,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:35:15,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148379 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:35:19,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:35:19,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855428 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:35:25,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:35:25,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150525 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:35:29,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:35:29,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131338 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:35:35,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:35:35,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254855 s, 161 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:35:39,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:35:39,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164545 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:35:45,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:35:45,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172254 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:35:49,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:35:49,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0023111 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14483::WARNING::2013-03-17 18:35:55,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:35:55,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:35:55,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:35:55,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766628 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:35:59,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:35:59,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915316 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:36:05,613::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:36:05,614::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:05,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:05,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159177 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:36:06,100::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:36:06,107::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:36:06,108::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:36:09,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:36:09,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144897 s, 283 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:36:15,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:15,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742231 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:36:19,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:36:19,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788479 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:36:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:25,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726021 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:36:29,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:36:29,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142603 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:36:36,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:36,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777439 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:36:39,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:36:39,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870694 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:36:46,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:46,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143697 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:36:49,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:36:49,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0398605 s, 103 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:36:56,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:36:56,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144595 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:36:59,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:00,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133773 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14484::WARNING::2013-03-17 18:37:06,143::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:37:06,147::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:37:06,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:06,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088568 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:10,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:10,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894616 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:37:16,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:37:16,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:16,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:16,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156164 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:37:16,607::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:37:16,613::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:37:16,614::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:20,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:20,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084242 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:37:26,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:26,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140159 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:30,110::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:37:30,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:37:30,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:30,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000830389 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:30,138::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000830389 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:37:30,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:37:30,139::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:37:30,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:37:30,141::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:30,142::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:30,142::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:30,143::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:30,143::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:30,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:30,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:30,549::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:37:30,550::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:37:30,552::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:37:30,554::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:37:30,554::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:37:30,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:30,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:30,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:30,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169534 s, 242 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:37:36,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:36,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018924 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:40,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:41,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799577 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:37:46,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:46,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139908 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:37:51,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:37:51,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143567 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:37:56,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:37:56,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087929 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:38:01,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:38:01,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144238 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:38:06,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:06,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182845 s, 224 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:38:11,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:38:11,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161584 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14485::WARNING::2013-03-17 18:38:16,643::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:38:16,649::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:38:16,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:16,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100496 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:38:21,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:38:21,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160113 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:38:26,652::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:38:26,654::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:26,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:26,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173806 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:38:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:38:27,140::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:38:27,145::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:38:31,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:38:31,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885269 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:38:37,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:37,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807775 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:38:41,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:38:41,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140591 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:38:47,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:47,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103652 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:38:51,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:38:51,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198735 s, 206 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:38:57,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:38:57,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800627 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:39:01,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:39:01,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223657 s, 183 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:07,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:07,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147127 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:39:11,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:39:11,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218559 s, 187 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:17,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:17,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142419 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:39:21,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:39:21,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142564 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14486::WARNING::2013-03-17 18:39:27,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:39:27,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:39:27,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:27,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882694 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:39:31,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:39:31,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213163 s, 192 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:39:37,186::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:39:37,187::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:39:37,188::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:37,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:37,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139269 s, 2.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:39:37,648::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:39:37,654::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:39:37,655::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:39:41,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:39:41,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214244 s, 191 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:47,616::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:39:47,617::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:39:47,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:47,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015839 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:47,664::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015839 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:39:47,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:39:47,665::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:39:47,667::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:39:47,668::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:39:47,668::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:39:47,669::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:39:47,669::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:39:47,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:39:47,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:48,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:48,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:39:48,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:39:48,113::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:39:48,114::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:39:48,114::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:39:48,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:48,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:48,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:48,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00695315 s, 589 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:39:51,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:39:51,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123514 s, 332 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:39:58,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:39:58,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778433 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:40:01,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:40:01,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192639 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:40:08,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:40:08,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147696 s, 277 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:40:11,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:40:11,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078369 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:40:18,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:40:18,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822251 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:40:21,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:40:21,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208454 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:40:28,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:40:28,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848713 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:40:31,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:40:32,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863491 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14487::WARNING::2013-03-17 18:40:37,680::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:40:37,687::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:40:38,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:40:38,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924888 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:40:42,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:40:42,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814081 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:40:47,691::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:40:47,692::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:40:48,108::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:40:48,114::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:40:48,116::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:40:48,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:40:48,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0091686 s, 447 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:40:52,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:40:52,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020867 s, 196 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:40:59,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:40:59,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153986 s, 266 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:41:02,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:41:02,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183479 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:41:09,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:41:09,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175071 s, 234 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:41:12,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:41:12,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140862 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:41:19,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:41:19,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160697 s, 255 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:41:22,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:41:22,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144854 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:41:29,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:41:29,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158098 s, 259 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:41:32,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:41:32,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083599 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:41:39,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:41:39,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158265 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:41:42,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:41:42,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197189 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14488::WARNING::2013-03-17 18:41:48,142::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:41:48,147::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:41:49,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:41:49,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151283 s, 271 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:41:52,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:41:52,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008179 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:41:58,149::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:41:58,150::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:41:58,558::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:41:58,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:41:58,566::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:41:59,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:41:59,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017253 s, 237 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:02,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:02,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101491 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:42:09,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:42:09,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147822 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:12,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:12,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150835 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:42:19,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:42:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110714 s, 370 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:22,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:22,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179217 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:42:29,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:42:29,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015666 s, 261 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:32,543::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:42:32,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:42:32,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:32,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00088916 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:32,580::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00088916 s, 2.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:42:32,581::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:42:32,581::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:42:32,583::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:42:32,585::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:42:32,586::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:42:32,586::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:42:32,587::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:42:32,587::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:42:32,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:33,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:33,037::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:42:33,038::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:42:33,041::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:42:33,042::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:42:33,043::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:42:33,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:33,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:33,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:33,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100726 s, 407 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:42:39,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:42:39,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163813 s, 250 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:43,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:43,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781117 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:42:50,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:42:50,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205153 s, 200 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:42:53,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:42:53,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837223 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14489::WARNING::2013-03-17 18:42:58,599::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:42:58,605::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:43:00,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:43:00,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800042 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:43:03,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:43:03,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852097 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:43:08,608::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:43:08,609::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:43:09,020::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:43:09,027::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:43:09,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:43:10,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:43:10,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00277345 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:43:13,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:43:13,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156725 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:43:20,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:43:20,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814848 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:43:23,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:43:23,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751947 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:43:30,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:43:30,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795626 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:43:33,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:43:33,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171664 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:43:40,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:43:40,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153519 s, 267 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:43:43,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:43:43,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805139 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:43:50,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:43:50,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185462 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:43:53,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:43:53,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143391 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:00,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:00,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124968 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:44:03,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:44:03,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753358 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14490::WARNING::2013-03-17 18:44:09,055::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:44:09,060::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:44:10,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:10,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906449 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:44:13,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:44:13,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911796 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:44:19,064::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:44:19,065::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:44:19,537::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:44:19,544::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:44:19,546::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:20,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:20,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00301663 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:44:23,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:44:23,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875969 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:30,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:30,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945577 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:44:33,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:44:33,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211489 s, 194 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:40,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:40,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835374 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:44:43,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:44:43,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207824 s, 197 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:50,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:44:50,982::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:44:50,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:51,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0169367 s, 121 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:51,048::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0169367 s, 121 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:44:51,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:44:51,048::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:44:51,050::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:44:51,051::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:51,051::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:51,052::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:51,052::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:51,053::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:51,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:51,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:51,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:44:51,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:44:51,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:44:51,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:44:51,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:44:51,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:51,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:44:51,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:44:51,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096672 s, 424 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:44:54,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:44:54,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196143 s, 209 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:45:01,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:45:01,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144954 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:45:04,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:45:04,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0955131 s, 42.9 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:45:11,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:45:12,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189377 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:45:14,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:45:14,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00282248 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14491::WARNING::2013-03-17 18:45:19,584::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:45:19,588::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:45:22,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:45:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842037 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:45:24,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:45:24,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155054 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:45:29,589::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:45:29,591::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:45:29,592::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:45:30,048::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:45:30,055::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:45:30,056::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:45:32,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:45:32,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798931 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:45:34,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:45:34,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916879 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:45:42,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:45:42,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150032 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:45:44,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:45:44,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147278 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:45:52,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:45:52,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787256 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:45:54,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:45:54,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157413 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:46:02,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:46:02,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742002 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:46:04,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:46:04,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263318 s, 156 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:46:12,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:46:12,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238362 s, 172 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:46:14,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:46:14,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937164 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:46:22,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:46:22,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191868 s, 213 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:46:24,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:46:24,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138215 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14492::WARNING::2013-03-17 18:46:30,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:46:30,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:46:32,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:46:32,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173832 s, 236 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:46:34,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:46:34,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827647 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:46:40,094::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:46:40,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:46:40,484::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:46:40,491::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:46:40,493::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:46:42,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:46:42,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154901 s, 264 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:46:44,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:46:44,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810497 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:46:52,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:46:52,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169968 s, 241 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:46:54,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:46:54,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154007 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:47:02,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:47:02,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197751 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:04,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:04,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214547 s, 191 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:47:12,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:47:13,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157682 s, 260 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:14,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795985 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:47:23,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:47:23,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139647 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:24,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:24,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741823 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:47:33,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:47:33,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841754 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:34,835::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:47:34,839::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:47:34,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:34,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00307559 s, 666 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:34,872::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00307559 s, 666 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:47:34,873::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:47:34,873::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:47:34,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:47:34,876::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:47:34,877::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:47:34,877::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:47:34,877::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:47:34,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:47:34,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:35,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:35,329::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:47:35,330::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:47:35,333::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:47:35,334::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:47:35,334::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:47:35,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:35,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:35,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:35,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193466 s, 212 kB/s\n'; <rc> = 0 >Thread-14493::WARNING::2013-03-17 18:47:40,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:47:40,525::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:47:43,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:47:43,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168389 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:45,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:45,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221158 s, 1.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:47:50,527::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:47:50,528::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:47:50,949::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:47:50,955::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:47:50,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:47:53,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:47:53,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016436 s, 249 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:47:55,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:47:55,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379909 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:48:03,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:48:03,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171765 s, 238 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:48:05,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:48:05,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145953 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:48:13,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:48:13,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782914 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:48:15,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:48:15,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138224 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:48:23,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:48:23,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167682 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:48:25,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:48:25,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792572 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:48:33,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:48:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122448 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:48:35,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:48:36,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911707 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:48:43,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:48:43,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206089 s, 199 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:48:46,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:48:46,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175977 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14494::WARNING::2013-03-17 18:48:50,984::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:48:50,991::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:48:53,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:48:53,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782222 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:48:56,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:48:56,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00502656 s, 815 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:49:00,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:49:00,995::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:49:01,460::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:49:01,467::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:49:01,468::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:03,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:03,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178367 s, 230 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:49:06,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:49:06,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991551 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:14,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:14,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00252862 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:49:16,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:49:16,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902527 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:24,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:24,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0284338 s, 144 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:49:26,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:49:26,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783513 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:34,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:34,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107321 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:49:36,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:49:36,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100342 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:44,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:44,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774949 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:49:46,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:49:46,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014053 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:54,323::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:49:54,324::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:49:54,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:54,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0166053 s, 123 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:54,378::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0166053 s, 123 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:49:54,379::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:49:54,379::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:49:54,381::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:49:54,384::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:54,385::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:54,385::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:54,385::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:54,386::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:54,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:54,815::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:49:54,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:49:54,819::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:49:54,820::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:49:54,820::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:49:54,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:55,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:49:55,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:49:55,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00909608 s, 450 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:49:56,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:49:56,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176039 s, 233 kB/s\n'; <rc> = 0 >Thread-14495::WARNING::2013-03-17 18:50:01,500::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:50:01,505::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:50:05,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:50:05,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091137 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:50:06,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:50:06,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149492 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:50:11,508::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:50:11,509::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:50:11,936::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:50:11,943::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:50:11,944::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:50:15,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:50:15,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761734 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:50:16,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:50:16,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077337 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:50:25,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:50:25,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826626 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:50:26,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:50:26,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853104 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:50:35,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:50:35,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141394 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:50:36,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:50:36,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265667 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:50:45,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:50:45,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146096 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:50:46,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:50:46,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796874 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:50:55,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:50:55,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117275 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:50:56,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:50:56,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108799 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:51:05,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:51:05,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689767 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:51:06,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:51:06,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144374 s, 284 kB/s\n'; <rc> = 0 >Thread-14496::WARNING::2013-03-17 18:51:11,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:51:11,977::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:51:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:51:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160404 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:51:16,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:51:16,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893383 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:51:21,979::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:51:21,980::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:51:21,982::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:51:22,413::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:51:22,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:51:22,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:51:25,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:51:25,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080206 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:51:26,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:51:26,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911178 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:51:35,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:51:36,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012034 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:51:36,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:51:36,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924794 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:51:46,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:51:46,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0311805 s, 131 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:51:46,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:51:46,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194832 s, 210 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:51:56,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:51:56,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834007 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:51:56,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:51:56,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147706 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:52:06,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:52:06,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147285 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:06,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:06,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129075 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:52:16,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:52:16,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777653 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:16,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:16,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204264 s, 201 kB/s\n'; <rc> = 0 >Thread-14497::WARNING::2013-03-17 18:52:22,464::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:52:22,468::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:52:26,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:52:26,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090684 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:26,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:27,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151635 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:52:32,470::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:52:32,471::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:52:32,891::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:52:32,898::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:52:32,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:52:36,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:52:36,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782077 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:37,016::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:52:37,018::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:52:37,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:37,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0200928 s, 102 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:37,084::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0200928 s, 102 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:52:37,084::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:52:37,085::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:52:37,086::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:52:37,087::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:52:37,088::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:52:37,088::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:52:37,089::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:52:37,089::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:52:37,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:37,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:37,472::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:52:37,473::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:52:37,476::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:52:37,477::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:52:37,478::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:52:37,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:37,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:37,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:37,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021906 s, 187 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:52:46,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:52:46,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00216925 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:47,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:47,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085949 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:52:56,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:52:56,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161152 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:52:57,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:52:57,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744845 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:53:06,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:53:06,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786582 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:53:07,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:53:08,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926769 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:53:16,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:53:16,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230108 s, 178 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:53:18,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:53:18,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200357 s, 204 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:53:26,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:53:26,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132157 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:53:28,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:53:28,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214564 s, 191 kB/s\n'; <rc> = 0 >Thread-14498::WARNING::2013-03-17 18:53:32,926::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:53:32,930::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:53:36,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:53:36,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198852 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:53:38,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:53:38,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0319263 s, 128 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:53:42,932::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:53:42,933::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:53:43,339::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:53:43,346::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:53:43,347::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:53:47,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:53:47,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00447846 s, 915 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:53:48,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:53:48,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862083 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:53:57,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:53:57,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179313 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:53:58,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:53:58,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164892 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:07,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:07,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243263 s, 168 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:54:08,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:54:08,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878978 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:17,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:17,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765898 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:54:18,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:54:18,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797692 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:27,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:27,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150729 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:54:28,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:54:28,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013845 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:37,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:37,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294319 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:54:38,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:54:38,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140548 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14499::WARNING::2013-03-17 18:54:43,377::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:54:43,381::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:54:47,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:47,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148782 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:54:48,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:54:48,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870632 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:54:53,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:54:53,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:54:53,798::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:54:53,805::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:54:53,806::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:57,619::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 18:54:57,622::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:54:57,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:57,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000784537 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:57,649::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000784537 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 18:54:57,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 18:54:57,650::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 18:54:57,652::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 18:54:57,653::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:57,654::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:57,654::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:57,654::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:57,655::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:57,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:58,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:58,092::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:54:58,093::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 18:54:58,095::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 18:54:58,097::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 18:54:58,097::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 18:54:58,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:58,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:54:58,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:54:58,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:54:58,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00528508 s, 775 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:54:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170416 s, 240 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:55:08,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:08,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:08,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013131 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:55:08,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807756 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:55:18,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:18,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:55:18,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234222 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:55:18,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107767 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:55:28,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:55:28,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861429 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:55:28,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:28,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143222 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:55:38,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:55:38,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143424 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:55:38,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:38,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013755 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:55:48,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:55:48,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145296 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:55:49,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:49,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978939 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14500::WARNING::2013-03-17 18:55:53,837::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:55:53,842::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:55:58,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:55:59,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895788 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:55:59,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:55:59,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819283 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:56:03,845::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:56:03,846::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:56:04,285::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:56:04,292::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:56:04,293::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:56:09,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:56:09,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187027 s, 219 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:56:09,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:56:09,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172282 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:56:19,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:56:19,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225962 s, 181 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:56:19,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:56:19,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836783 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:56:29,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:56:29,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0325891 s, 126 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:56:29,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:56:29,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759886 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:56:39,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:56:39,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229052 s, 179 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:56:39,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:56:39,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116125 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:56:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:56:49,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219544 s, 187 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:56:49,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:56:49,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016044 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:56:59,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:56:59,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023867 s, 172 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:56:59,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:56:59,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162079 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14501::WARNING::2013-03-17 18:57:04,318::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:57:04,322::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 18:57:09,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:09,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213393 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:57:09,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:57:09,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208274 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:57:14,323::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 18:57:14,324::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:57:14,325::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:57:14,769::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:57:14,776::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:57:14,777::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:19,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:19,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828116 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:57:19,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:57:19,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839529 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:29,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:29,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844188 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:57:29,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:57:29,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773392 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:39,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 18:57:39,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:57:39,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:39,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134376 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:39,581::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134376 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 18:57:39,582::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 18:57:39,582::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 18:57:39,584::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 18:57:39,585::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:39,585::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:39,586::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:39,586::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:39,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:39,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 18:57:39,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:57:39,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160122 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:40,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:40,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 18:57:40,029::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 18:57:40,032::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 18:57:40,033::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 18:57:40,034::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 18:57:40,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:40,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:40,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:40,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134654 s, 304 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:57:49,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:57:50,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773738 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:57:50,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:57:50,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751043 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:58:00,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:58:00,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896703 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:58:00,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:58:00,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163334 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:58:10,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:58:10,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168342 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:58:10,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:58:10,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120128 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14502::WARNING::2013-03-17 18:58:14,806::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:58:14,812::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:58:20,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:58:20,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0322821 s, 127 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:58:20,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:58:20,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114053 s, 3.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:58:24,816::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:58:24,817::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:58:25,224::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:58:25,231::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:58:25,232::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:58:30,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:58:30,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850888 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:58:30,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:58:30,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765484 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:58:40,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:58:40,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774807 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:58:40,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:58:40,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141942 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:58:50,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:58:50,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169196 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:58:50,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:58:50,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161887 s, 253 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:59:00,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:59:00,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917697 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:59:00,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:59:00,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838405 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:59:10,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:59:10,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765106 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:59:10,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:59:10,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088707 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:59:20,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:59:20,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789783 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:59:20,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:59:20,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119896 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14503::WARNING::2013-03-17 18:59:25,268::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 18:59:25,275::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 18:59:30,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:59:30,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859457 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:59:30,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:59:30,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077214 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 18:59:35,278::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 18:59:35,280::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 18:59:35,699::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 18:59:35,706::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 18:59:35,708::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 18:59:40,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:59:40,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840002 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:59:40,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:59:40,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146972 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 18:59:50,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 18:59:50,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167957 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 18:59:50,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 18:59:51,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819917 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:00:01,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:01,045::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:00:01,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:00:01,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:00:01,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124989 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:01,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00683052 s, 300 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:01,139::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00683052 s, 300 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:00:01,139::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:00:01,140::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:00:01,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:00:01,141::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:00:01,142::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:00:01,143::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:00:01,144::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:00:01,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:00:01,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:01,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:01,703::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:00:01,704::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:00:01,706::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:00:01,707::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:00:01,708::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:00:01,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:02,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:02,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:02,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105297 s, 389 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:00:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:00:11,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145332 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:12,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:12,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149806 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:00:21,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:00:21,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170207 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:22,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:22,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149874 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:00:31,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:00:31,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997314 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:32,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:32,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168388 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14504::WARNING::2013-03-17 19:00:35,734::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:00:35,738::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:00:41,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:00:41,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0311795 s, 131 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:42,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833933 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:00:45,740::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:00:45,741::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:00:46,155::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:00:46,162::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:00:46,163::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:00:51,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:00:51,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920768 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:00:52,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:00:52,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713403 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:01:01,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:01:01,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142487 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:01:02,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:01:02,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162299 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:01:11,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:01:11,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00462663 s, 885 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:01:12,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:01:12,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015757 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:01:21,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:01:21,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148818 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:01:22,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:01:22,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779922 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:01:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:01:31,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899419 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:01:33,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:01:33,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152399 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:01:41,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:01:41,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131229 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:01:43,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:01:43,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141365 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14505::WARNING::2013-03-17 19:01:46,192::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:01:46,195::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:01:51,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:01:51,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100433 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:01:53,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:01:53,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875846 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:01:56,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:01:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:01:56,622::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:01:56,628::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:01:56,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:01,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:01,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774721 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:02:03,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:02:03,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992163 s, 413 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:11,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:11,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123177 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:02:13,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:02:13,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369761 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:21,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:21,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014802 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:02:23,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:02:23,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194998 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:31,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:31,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898286 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:02:33,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:02:33,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980127 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:41,866::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:02:41,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:02:41,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0202964 s, 101 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:41,927::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0202964 s, 101 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:02:41,927::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:02:41,927::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:02:41,929::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:02:41,930::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:41,931::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:41,931::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:41,932::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:41,933::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:42,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:42,381::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:02:42,382::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:02:42,384::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:02:42,385::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:02:42,386::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:02:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:42,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:42,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:42,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116089 s, 353 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:02:43,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:02:43,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135508 s, 302 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:02:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:02:52,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000663785 s, 6.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:02:53,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:02:53,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946557 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14506::WARNING::2013-03-17 19:02:56,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:02:56,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:03:02,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:03:02,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113995 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:03:03,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:03:03,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825868 s, 496 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:03:06,672::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:03:06,674::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:03:06,675::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:03:07,189::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:03:07,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:03:07,198::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:03:12,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:03:12,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837705 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:03:13,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:03:13,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104581 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:03:22,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:03:22,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777303 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:03:23,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:03:23,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013001 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:03:32,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:03:33,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820262 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:03:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:03:33,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153924 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:03:43,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:03:43,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773284 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:03:44,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:03:44,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165508 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:03:53,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:03:53,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152466 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:03:54,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:03:54,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191118 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:04:03,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:04:03,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149294 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:04:04,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:04:04,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146493 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14507::WARNING::2013-03-17 19:04:07,233::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:04:07,237::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:04:13,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:04:13,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760617 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:04:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:04:14,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185538 s, 2.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:04:17,239::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:04:17,240::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:04:17,654::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:04:17,661::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:04:17,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:04:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:04:23,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139721 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:04:24,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:04:24,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142361 s, 288 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:04:33,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:04:33,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210811 s, 194 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:04:34,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:04:34,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150088 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:04:43,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:04:43,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00902986 s, 454 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:04:44,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:04:44,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820483 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:04:53,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:04:53,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0414716 s, 98.8 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:04:54,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:04:54,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777091 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:05:03,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:05:03,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136169 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:04,723::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:05:04,725::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:05:04,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:04,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152546 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:04,765::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152546 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:05:04,766::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:05:04,766::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:05:04,768::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:05:04,769::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:05:04,769::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:05:04,770::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:05:04,770::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:05:04,770::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:05:04,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:05,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:05,171::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:05:05,173::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:05:05,175::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:05:05,177::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:05:05,177::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:05:05,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:05,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:05,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:05,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143546 s, 285 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:05:13,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:05:13,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756816 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:15,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823866 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14508::WARNING::2013-03-17 19:05:17,688::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:05:17,691::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:05:23,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:05:23,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862748 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:25,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:25,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0093369 s, 439 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:05:27,693::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:05:27,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:05:28,129::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:05:28,136::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:05:28,137::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:05:33,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:05:33,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861539 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:35,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:35,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811008 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:05:43,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:05:43,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747751 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:45,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:45,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164295 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:05:53,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:05:53,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134274 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:05:55,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:05:55,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150597 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:06:03,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:06:03,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801623 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:06:06,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:06:06,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148681 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:06:13,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:06:13,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171053 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:06:16,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:06:16,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147543 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:06:23,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:06:23,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770115 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:06:26,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:06:26,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096785 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14509::WARNING::2013-03-17 19:06:28,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:06:28,167::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:06:33,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:06:34,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822337 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:06:36,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:06:36,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793653 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:06:38,170::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:06:38,172::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:06:38,598::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:06:38,608::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:06:38,609::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:06:44,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:06:44,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0386051 s, 106 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:06:46,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:06:46,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153443 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:06:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:06:54,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00617399 s, 663 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:06:56,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:06:56,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154276 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:04,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:04,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00978403 s, 419 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:07:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:07:06,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777965 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:14,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:14,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135466 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:07:16,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:07:16,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105587 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:24,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:24,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00384299 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:07:26,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:07:26,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728444 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:34,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:34,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169954 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:07:36,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:07:36,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934019 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14510::WARNING::2013-03-17 19:07:38,648::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:07:38,652::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:07:44,362::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:07:44,364::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:07:44,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:44,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000786895 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:44,397::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000786895 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:07:44,398::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:07:44,398::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:07:44,399::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:07:44,401::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:44,401::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:44,402::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:44,402::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:44,403::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:44,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:44,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:44,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:44,837::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:07:44,839::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:07:44,840::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:07:44,841::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:07:44,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:45,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:07:45,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:45,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013972 s, 293 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:07:46,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:07:46,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768981 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:07:48,654::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:07:48,655::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:07:49,055::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:07:49,061::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:07:49,063::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:07:55,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:07:55,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891196 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:07:56,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:07:56,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135372 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:08:05,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:08:05,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152184 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:08:06,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:08:07,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0311293 s, 132 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:08:15,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:08:15,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807075 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:08:17,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:08:17,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924187 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:08:25,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:08:25,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142016 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:08:27,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:08:27,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761355 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:08:35,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:08:35,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130764 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:08:37,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:08:37,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780383 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:08:45,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:08:45,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189587 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:08:47,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:08:47,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314786 s, 1.3 MB/s\n'; <rc> = 0 >Thread-14511::WARNING::2013-03-17 19:08:49,090::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:08:49,095::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:08:55,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:08:55,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809222 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:08:57,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:08:57,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104478 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:08:59,097::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:08:59,099::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:08:59,100::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:08:59,531::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:08:59,537::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:08:59,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:09:05,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:09:05,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821718 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:09:07,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:09:07,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964759 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:09:15,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:09:15,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079104 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:09:17,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:09:17,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898324 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:09:25,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:09:25,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755228 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:09:27,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:09:27,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797876 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:09:35,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:09:35,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141653 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:09:37,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:09:37,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130729 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:09:45,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:09:45,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138435 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:09:47,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:09:48,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185856 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:09:55,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:09:55,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148274 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:09:58,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:09:58,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105761 s, 3.9 MB/s\n'; <rc> = 0 >Thread-14512::WARNING::2013-03-17 19:09:59,567::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:09:59,570::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:10:05,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:10:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951021 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:08,178::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:10:08,179::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:10:08,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:08,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00108777 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:08,207::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00108777 s, 1.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:10:08,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:10:08,208::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:10:08,209::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:10:08,211::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:10:08,211::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:10:08,212::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:10:08,212::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:10:08,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:10:08,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:08,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:08,644::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:10:08,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:10:08,648::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:10:08,649::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:10:08,650::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:10:08,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:09,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:09,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:09,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106752 s, 384 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:10:09,572::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:10:09,574::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:10:09,993::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:10:10,000::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:10:10,001::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:10:15,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:10:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974035 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:19,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:19,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154268 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:10:25,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:10:25,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868531 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:29,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:29,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885791 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:10:35,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:10:35,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134538 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:39,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:39,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737634 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:10:45,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:10:45,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909604 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:49,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:49,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968674 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:10:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:10:55,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160348 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:10:59,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:10:59,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875128 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:11:06,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:11:06,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124472 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:11:09,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:11:09,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732119 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14513::WARNING::2013-03-17 19:11:10,028::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:11:10,032::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:11:16,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:11:16,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744064 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:11:19,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:11:19,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771074 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:11:20,035::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:11:20,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:11:20,485::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:11:20,492::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:11:20,493::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:11:26,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:11:26,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161247 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:11:29,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:11:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900826 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:11:36,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:11:36,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905087 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:11:40,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:11:40,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167112 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:11:46,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:11:46,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147784 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:11:50,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:11:50,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924872 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:11:56,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:11:56,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016113 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:12:00,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:00,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179987 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:06,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:06,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0567168 s, 72.2 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:12:10,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:10,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829023 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:16,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:16,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171226 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:12:20,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:20,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00635375 s, 645 kB/s\n'; <rc> = 0 >Thread-14514::WARNING::2013-03-17 19:12:20,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:12:20,533::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:12:26,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:26,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792342 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:12:30,538::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:12:30,539::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:30,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:30,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161336 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:12:31,070::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:12:31,075::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:12:31,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:36,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:36,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995513 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:12:40,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:40,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00229929 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:46,492::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:12:46,493::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:12:46,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:46,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00190002 s, 1.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:46,534::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00190002 s, 1.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:12:46,534::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:12:46,534::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:12:46,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:12:46,537::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:46,537::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:46,538::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:46,538::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:46,538::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:46,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:46,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:46,972::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:12:46,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:12:46,978::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:12:46,979::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:12:46,980::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:12:46,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:47,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:47,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:47,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132104 s, 310 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:12:50,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:12:50,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144656 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:12:57,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:12:57,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281844 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:13:00,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:13:00,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175484 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:13:07,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:13:07,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128891 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:13:10,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:13:10,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164291 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:13:17,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:13:17,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163486 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:13:20,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:13:21,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167133 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:13:27,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:13:27,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795704 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:13:31,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-14515::WARNING::2013-03-17 19:13:31,110::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:13:31,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:13:31,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240685 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:13:37,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:13:37,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946693 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:13:41,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:13:41,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 19:13:41,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:13:41,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00925355 s, 443 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:13:41,655::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:13:41,660::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:13:41,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:13:47,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:13:47,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155473 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:13:51,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:13:51,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144697 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:13:57,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:13:57,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862291 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:14:01,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:01,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106884 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:14:07,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:14:07,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158766 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:14:11,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:11,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896453 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:14:17,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:14:17,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709915 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:14:21,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:21,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142636 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:14:27,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:14:27,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144651 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:14:31,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:31,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910278 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:14:37,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:14:37,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149217 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14516::WARNING::2013-03-17 19:14:41,690::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:14:41,696::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:14:41,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:41,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817975 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:14:47,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:14:47,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795138 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:14:51,698::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:14:51,699::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:14:51,701::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:51,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:14:51,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202193 s, 203 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:14:52,193::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:14:52,199::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:14:52,200::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:14:57,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:14:57,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743356 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:01,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:01,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849105 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:15:07,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:15:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119143 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:11,967::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:15:11,968::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:15:11,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:11,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000830081 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:11,998::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000830081 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:15:11,998::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:15:11,998::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:15:12,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:15:12,001::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:15:12,002::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:15:12,002::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:15:12,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:15:12,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:15:12,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:12,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:12,431::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:15:12,433::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:15:12,435::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:15:12,436::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:15:12,437::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:15:12,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:12,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:12,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:12,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968942 s, 423 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:15:17,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:15:18,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0303807 s, 135 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:22,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:22,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147391 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:15:28,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:15:28,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144511 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:32,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:33,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832162 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:15:38,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:15:38,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161473 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:15:43,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:43,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161696 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:15:48,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:15:48,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00431752 s, 949 kB/s\n'; <rc> = 0 >Thread-14517::WARNING::2013-03-17 19:15:52,225::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:15:52,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:15:53,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:15:53,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799025 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:15:58,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:15:58,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765354 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:16:02,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:16:02,232::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:16:02,628::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:16:02,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:16:02,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:16:03,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:16:03,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197102 s, 208 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:16:08,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:16:08,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124846 s, 328 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:16:13,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:16:13,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793845 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:16:18,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:16:18,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904546 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:16:23,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:16:23,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150523 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:16:28,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:16:28,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907002 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:16:33,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:16:33,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906235 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:16:38,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:16:38,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779585 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:16:43,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:16:43,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0309103 s, 133 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:16:48,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:16:48,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905328 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:16:53,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:16:53,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144881 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:16:58,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:16:58,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936908 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14518::WARNING::2013-03-17 19:17:02,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:17:02,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:17:03,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:17:03,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807175 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:08,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:08,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141403 s, 2.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:17:12,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:17:12,679::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:17:13,100::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:17:13,106::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:17:13,108::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:17:13,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:17:13,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00940203 s, 436 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:18,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:18,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140297 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:17:24,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:17:24,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016553 s, 247 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:28,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:28,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149247 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:17:34,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:17:34,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00769249 s, 532 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:38,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:38,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915746 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:17:44,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:17:44,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132757 s, 309 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:48,668::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:17:48,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:17:48,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:48,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0140224 s, 146 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:48,710::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0140224 s, 146 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:17:48,715::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:17:48,716::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:17:48,718::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:17:48,719::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:17:48,719::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:17:48,720::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:17:48,720::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:17:48,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:17:48,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:49,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:49,150::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:17:49,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:17:49,153::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:17:49,155::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:17:49,155::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:17:49,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:49,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:49,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:49,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798987 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:17:54,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:17:54,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145046 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:17:59,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:17:59,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159327 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:18:04,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:18:04,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130145 s, 315 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:18:09,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:18:09,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129884 s, 3.2 MB/s\n'; <rc> = 0 >Thread-14519::WARNING::2013-03-17 19:18:13,135::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:18:13,138::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:18:14,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:18:14,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936658 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:18:19,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:18:19,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113371 s, 361 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:18:23,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:18:23,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:18:23,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:18:23,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:18:23,595::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:18:24,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:18:24,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169796 s, 241 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:18:29,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:18:29,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166218 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:18:34,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:18:34,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143939 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:18:39,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:18:39,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00773735 s, 529 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:18:44,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:18:44,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00501833 s, 816 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:18:49,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:18:49,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138538 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:18:54,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:18:54,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766114 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:18:59,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:18:59,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213565 s, 192 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:19:04,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:19:04,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142913 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:19:09,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:19:09,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104246 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:19:14,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:19:14,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851289 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:19:19,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:19:19,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143492 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14520::WARNING::2013-03-17 19:19:23,631::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:19:23,635::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:19:25,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:19:25,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157076 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:19:29,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:19:30,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830799 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:19:33,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:19:33,639::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:19:34,047::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:19:34,054::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:19:34,055::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:19:35,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:19:35,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995158 s, 412 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:19:40,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:19:40,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000669893 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:19:45,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:19:45,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130054 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:19:50,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:19:50,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746968 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:19:55,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:19:55,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135654 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:20:00,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:20:00,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151189 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:05,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:05,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141453 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:20:10,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:20:10,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835398 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:15,467::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:20:15,469::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:20:15,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:15,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000760801 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:15,498::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000760801 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:20:15,499::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:20:15,499::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:20:15,501::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:20:15,502::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:15,502::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:15,503::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:15,503::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:15,504::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:15,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:15,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:15,933::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:15,934::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:20:15,936::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:20:15,938::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:20:15,938::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:20:15,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:16,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:16,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:16,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139114 s, 294 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:20:20,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:20:20,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010103 s, 405 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:26,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:26,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013583 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:20:30,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:20:30,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149758 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14521::WARNING::2013-03-17 19:20:34,081::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:20:34,085::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:20:36,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:36,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763747 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:20:40,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:20:40,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824136 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:20:44,086::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:20:44,088::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:20:44,089::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:20:44,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:20:44,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:20:44,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:20:46,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:46,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194161 s, 211 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:20:50,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:20:50,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902659 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:20:56,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:20:56,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822255 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:21:00,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:21:00,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143844 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:21:06,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:21:06,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802389 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:21:10,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:21:10,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880193 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:21:16,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:21:16,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083597 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:21:20,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:21:20,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159235 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:21:26,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:21:26,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815863 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:21:30,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:21:30,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147387 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:21:37,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:21:37,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014126 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:21:40,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:21:40,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012597 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14522::WARNING::2013-03-17 19:21:44,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:21:44,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:21:47,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:21:47,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0092379 s, 443 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:21:50,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:21:50,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854522 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:21:54,601::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:21:54,602::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:21:55,064::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:21:55,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:21:55,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:21:57,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:21:57,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168526 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:00,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:00,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075296 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:22:07,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:22:07,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861185 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:10,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:10,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933607 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:22:17,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:22:17,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978087 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:20,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:20,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139985 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:22:27,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:22:27,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783451 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:30,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:30,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870181 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:22:37,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:22:37,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741671 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:40,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:40,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140769 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:22:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:22:47,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017542 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:50,804::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:22:50,806::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:22:50,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:50,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133319 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:50,847::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133319 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:22:50,848::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:22:50,848::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:22:50,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:22:50,853::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:22:50,854::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:22:50,854::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:22:50,854::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:22:50,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:22:50,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:51,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:51,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:22:51,307::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:22:51,311::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:22:51,312::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:22:51,312::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:22:51,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:51,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:22:51,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:22:51,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178461 s, 230 kB/s\n'; <rc> = 0 >Thread-14523::WARNING::2013-03-17 19:22:55,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:22:55,104::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:22:57,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:22:57,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028881 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:23:01,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:23:01,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109755 s, 3.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:23:05,106::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:23:05,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:23:05,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:23:05,509::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:23:05,511::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:23:07,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:23:07,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016035 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:23:11,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:23:11,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928267 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:23:17,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:23:17,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843642 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:23:21,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:23:21,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745719 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:23:27,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:23:27,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013214 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:23:31,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:23:31,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243727 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:23:37,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:23:37,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793694 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:23:41,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:23:41,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145385 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:23:48,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:23:48,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947604 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:23:51,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:23:51,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143411 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:23:58,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:23:58,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001292 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:24:02,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:24:02,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076885 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14524::WARNING::2013-03-17 19:24:05,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:24:05,549::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:24:08,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:24:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840426 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:24:12,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:24:12,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144976 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:24:15,552::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:24:15,553::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:24:16,024::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:24:16,031::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:24:16,032::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:24:18,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:24:18,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203668 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:24:22,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:24:22,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120955 s, 339 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:24:28,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:24:28,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259849 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:24:32,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:24:32,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789011 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:24:38,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:24:38,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240342 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:24:42,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:24:42,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00989002 s, 414 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:24:48,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:24:48,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760786 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:24:52,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:24:52,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141432 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:24:58,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:24:58,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732069 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:25:02,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:25:02,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152996 s, 268 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:08,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:08,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007932 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:25:12,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:25:12,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767295 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14525::WARNING::2013-03-17 19:25:16,060::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:25:16,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:25:18,738::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:25:18,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:25:18,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:18,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00181275 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:18,776::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00181275 s, 1.1 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:25:18,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:25:18,777::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:25:18,779::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:25:18,781::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:18,781::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:18,782::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:18,782::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:18,783::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:18,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:19,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:19,197::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:19,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:25:19,203::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:25:19,204::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:25:19,205::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:25:19,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:19,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:19,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:19,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00682068 s, 601 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:25:22,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:25:22,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970735 s, 422 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:25:26,072::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:25:26,074::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:25:26,517::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:25:26,524::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:25:26,525::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:25:29,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:29,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915455 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:25:32,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:25:32,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159562 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:39,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:39,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760303 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:25:42,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:25:42,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108212 s, 379 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:49,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:49,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162982 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:25:52,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:25:52,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903723 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:25:59,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:25:59,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169164 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:26:02,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:26:02,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138575 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:26:09,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:26:09,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136856 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:26:12,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:26:12,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774988 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:26:20,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:26:20,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141592 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:26:22,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:26:22,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138746 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14526::WARNING::2013-03-17 19:26:26,564::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:26:26,568::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:26:30,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:26:30,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824519 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:26:32,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:26:32,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193064 s, 212 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:26:36,577::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:26:36,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:26:36,580::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:26:37,016::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:26:37,022::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:26:37,023::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:26:40,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:26:40,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167352 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:26:42,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:26:42,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150493 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:26:50,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:26:50,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838176 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:26:52,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:26:52,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794384 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:27:00,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:27:00,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755841 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:02,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:02,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872687 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:27:10,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:27:10,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156781 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:12,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:13,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765608 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:27:20,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:27:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150998 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:23,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:23,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130451 s, 314 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:27:30,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:27:30,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151889 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:33,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:33,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074936 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14527::WARNING::2013-03-17 19:27:37,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:27:37,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:27:40,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:27:40,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846616 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:43,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:43,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921453 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:27:47,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:27:47,062::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:27:47,524::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:27:47,531::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:27:47,532::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:27:50,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:27:50,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137014 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:53,165::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:27:53,166::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:27:53,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:53,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0148559 s, 138 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:53,218::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0148559 s, 138 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:27:53,218::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:27:53,219::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:27:53,220::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:27:53,221::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:27:53,222::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:27:53,223::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:27:53,223::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:27:53,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:27:53,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:53,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:53,637::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:27:53,638::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:27:53,640::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:27:53,641::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:27:53,642::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:27:53,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:54,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:27:54,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:27:54,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167124 s, 245 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:28:00,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:28:00,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783952 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:28:04,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:28:04,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101581 s, 403 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:28:10,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:28:10,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832582 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:28:14,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:28:14,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151257 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:28:20,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:28:21,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085285 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:28:24,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:28:24,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107653 s, 380 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:28:31,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:28:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779858 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:28:34,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:28:34,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152634 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:28:41,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:28:41,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144713 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:28:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:28:44,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899918 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14528::WARNING::2013-03-17 19:28:47,559::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:28:47,563::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:28:51,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:28:51,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812162 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:28:54,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:28:54,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747621 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:28:57,565::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:28:57,566::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:28:58,011::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:28:58,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:28:58,019::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:29:01,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:29:01,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995794 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:29:04,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:29:04,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107641 s, 381 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:29:11,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:29:11,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174992 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:29:14,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:29:14,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118364 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:29:21,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:29:21,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790612 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:29:24,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:29:24,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113974 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:29:31,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:29:31,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163605 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:29:34,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:29:34,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741365 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:29:41,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:29:41,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761547 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:29:44,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:29:44,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159768 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:29:51,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:29:51,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156166 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:29:54,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:29:54,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153667 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14529::WARNING::2013-03-17 19:29:58,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:29:58,057::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:30:01,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:01,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00214882 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:30:04,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:30:04,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109736 s, 373 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:30:08,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:30:08,062::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:30:08,517::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:30:08,524::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:30:08,526::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:11,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:11,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122088 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:30:14,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:30:14,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814127 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:21,856::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:30:21,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:30:21,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:21,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015039 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:21,904::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015039 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:30:21,905::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:30:21,905::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:30:21,907::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:30:21,907::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:21,908::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:21,909::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:21,909::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:21,909::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:21,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:22,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:22,335::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:30:22,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:30:22,339::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:30:22,340::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:30:22,340::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:30:22,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:22,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:22,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:22,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162136 s, 253 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:30:24,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:30:24,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163313 s, 251 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:32,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:32,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816193 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:30:34,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:30:34,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191466 s, 214 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:42,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:42,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738655 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:30:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:30:44,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149686 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:30:52,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:30:53,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748149 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:30:54,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:30:54,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150638 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:31:03,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:31:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166153 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:31:04,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:31:04,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146217 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14530::WARNING::2013-03-17 19:31:08,553::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:31:08,557::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:31:13,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:31:13,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810517 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:31:14,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:31:15,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877088 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:31:18,559::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:31:18,560::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:31:18,998::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:31:19,005::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:31:19,006::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:31:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:31:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812128 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:31:25,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:31:25,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154911 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:31:33,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:31:33,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143385 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:31:35,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:31:35,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135023 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:31:43,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:31:43,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166182 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:31:45,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:31:45,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865163 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:31:53,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:31:53,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143836 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:31:55,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:31:55,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157073 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:32:03,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:32:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823945 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:05,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:05,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145697 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:32:13,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:32:13,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288511 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:15,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:15,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825464 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14531::WARNING::2013-03-17 19:32:19,051::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:32:19,055::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:32:23,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:32:23,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786818 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:25,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:25,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796695 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:32:29,057::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:32:29,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:32:29,059::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:32:29,491::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:32:29,499::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:32:29,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:32:33,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:32:33,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162425 s, 252 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:35,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:35,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133455 s, 307 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:32:43,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:32:43,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145002 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:45,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:45,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150994 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:32:53,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:32:53,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877966 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:55,425::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:32:55,427::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:32:55,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:55,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0115643 s, 177 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:55,468::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0115643 s, 177 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:32:55,469::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:32:55,469::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:32:55,471::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:32:55,473::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:32:55,474::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:32:55,474::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:32:55,474::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:32:55,475::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:32:55,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:55,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:55,904::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:32:55,905::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:32:55,907::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:32:55,909::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:32:55,909::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:32:55,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:56,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:32:56,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:32:56,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00950198 s, 431 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:33:04,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:33:04,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906569 s, 452 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:33:06,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:33:06,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084511 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:33:14,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:33:14,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145408 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:33:16,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:33:16,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183881 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:33:24,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:33:24,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918285 s, 446 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:33:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:33:26,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00390288 s, 1.0 MB/s\n'; <rc> = 0 >Thread-14532::WARNING::2013-03-17 19:33:29,529::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:33:29,534::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:33:34,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:33:34,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843937 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:33:36,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:33:36,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0048669 s, 842 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:33:39,536::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:33:39,537::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:33:39,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:33:39,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:33:39,989::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:33:44,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:33:44,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110121 s, 372 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:33:46,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:33:46,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094012 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:33:54,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:33:54,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805549 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:33:56,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:33:56,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137315 s, 298 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:34:04,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:34:04,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088995 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:34:06,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:34:06,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156911 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:34:14,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:34:14,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105498 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:34:16,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:34:16,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120081 s, 341 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:34:24,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:34:24,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923812 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:34:26,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:34:26,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014522 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:34:34,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:34:34,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138762 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:34:36,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:34:36,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00775067 s, 528 kB/s\n'; <rc> = 0 >Thread-14533::WARNING::2013-03-17 19:34:40,016::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:34:40,019::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:34:44,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:34:44,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166802 s, 246 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:34:46,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:34:46,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145749 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:34:50,021::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:34:50,023::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:34:50,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:34:50,488::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:34:50,489::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:34:54,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:34:54,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707782 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:34:56,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:34:56,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789579 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:04,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:04,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104266 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:35:06,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:35:06,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015411 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:15,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:15,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814921 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:35:16,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:35:16,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868602 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:25,097::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:35:25,098::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:35:25,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:25,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134923 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:25,131::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134923 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:35:25,132::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:35:25,132::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:35:25,134::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:35:25,135::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:35:25,136::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:35:25,136::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:35:25,136::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:35:25,137::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:35:25,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:25,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:25,561::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:35:25,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:35:25,565::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:35:25,566::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:35:25,567::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:35:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:25,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:25,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:25,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00712165 s, 575 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:35:26,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:35:27,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181561 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:36,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156356 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:35:37,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:35:37,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148787 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:35:46,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:46,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765804 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:35:47,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:35:47,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169245 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14534::WARNING::2013-03-17 19:35:50,518::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:35:50,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:35:56,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:35:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779664 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:35:57,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:35:57,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693955 s, 5.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:36:00,523::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:36:00,524::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:36:00,938::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:36:00,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:36:00,946::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:36:06,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:36:06,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794761 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:36:07,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:36:07,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289019 s, 142 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:36:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:36:16,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131842 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:36:17,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:36:17,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928154 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:36:26,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:36:26,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896179 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:36:27,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:36:27,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836987 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:36:36,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:36:36,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131554 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:36:37,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:36:37,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013193 s, 310 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:36:46,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:36:46,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145294 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:36:47,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:36:47,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129428 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:36:56,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:36:56,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969216 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:36:57,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:36:57,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122166 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14535::WARNING::2013-03-17 19:37:00,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:37:00,974::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:37:06,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:37:06,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106296 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:07,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:07,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079136 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:37:10,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:37:10,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:37:11,434::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:37:11,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:37:11,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:37:16,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:37:16,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070802 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:17,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:17,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909055 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:37:26,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:37:26,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801851 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:27,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:27,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077665 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:37:36,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:37:36,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726651 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:37,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:37,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146447 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:37:46,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:37:47,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801036 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:47,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:47,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108959 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:37:57,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:37:57,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144959 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:57,673::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:37:57,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:37:57,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:57,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00202719 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:57,708::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00202719 s, 1.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:37:57,708::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:37:57,708::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:37:57,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:37:57,711::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:37:57,712::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:37:57,712::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:37:57,713::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:37:57,713::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:37:57,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:58,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:58,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:37:58,147::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:37:58,149::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:37:58,150::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:37:58,151::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:37:58,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:58,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:37:58,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:37:58,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026869 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:38:07,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:38:07,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793406 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:38:08,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:38:08,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993259 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14536::WARNING::2013-03-17 19:38:11,471::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:38:11,477::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:38:17,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:38:17,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076165 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:38:18,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:38:18,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918082 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:38:21,479::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:38:21,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:38:21,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:38:21,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:38:21,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:38:21,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:38:27,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:38:27,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850324 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:38:28,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:38:28,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130307 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:38:37,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:38:37,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846523 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:38:38,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:38:38,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00522325 s, 784 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:38:47,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:38:47,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00210375 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:38:48,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:38:48,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835072 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:38:57,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:38:57,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847692 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:38:58,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:38:58,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829105 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:39:07,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:39:07,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774281 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:39:08,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:39:08,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019493 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:39:17,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:39:17,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016762 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:39:18,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:39:18,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132536 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14537::WARNING::2013-03-17 19:39:21,929::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:39:21,934::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:39:27,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:39:27,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760171 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:39:28,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:39:28,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125558 s, 3.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:39:31,936::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:39:31,937::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:39:32,370::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:39:32,377::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:39:32,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:39:37,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:39:37,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087614 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:39:38,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:39:39,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091741 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:39:47,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:39:47,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848496 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:39:49,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:39:49,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079777 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:39:57,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:39:58,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169257 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:39:59,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:39:59,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819435 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:08,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:08,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187907 s, 218 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:40:09,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:40:09,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740968 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:18,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:18,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851665 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:40:19,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:40:19,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148774 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:28,245::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:40:28,248::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:40:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:28,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136407 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:28,282::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136407 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:40:28,283::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:40:28,283::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:40:28,285::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:40:28,286::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:28,287::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:28,288::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:28,288::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:28,289::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:28,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:28,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:28,726::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:28,727::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:40:28,730::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:40:28,732::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:40:28,733::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:40:28,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:29,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:29,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:29,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108597 s, 377 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:40:29,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:40:29,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00654493 s, 626 kB/s\n'; <rc> = 0 >Thread-14538::WARNING::2013-03-17 19:40:32,412::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:40:32,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:40:39,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:39,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954554 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:40:39,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:40:39,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174492 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:40:42,418::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:40:42,419::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:40:42,862::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:40:42,869::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:40:42,871::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:40:49,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:40:49,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:49,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00771375 s, 531 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:40:49,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000688811 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:40:59,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:40:59,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:40:59,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128871 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:40:59,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155439 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:09,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:09,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:09,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779802 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:09,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168584 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:19,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:19,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105648 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:19,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00348847 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:29,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231448 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:29,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113108 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:40,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:40,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284558 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:41:40,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123712 s, 331 kB/s\n'; <rc> = 0 >Thread-14539::WARNING::2013-03-17 19:41:42,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:41:42,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:41:50,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:41:50,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:41:50,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113485 s, 361 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:41:50,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278527 s, 1.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:41:52,911::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:41:52,912::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:41:53,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:41:53,415::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:41:53,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:42:00,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:00,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:42:00,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180726 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:42:00,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117962 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:42:10,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:10,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:42:10,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113575 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:42:10,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914387 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:42:20,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:20,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:42:20,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131228 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:42:20,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106267 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:42:30,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:30,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:42:30,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862557 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:42:30,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101038 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:42:40,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:40,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:42:40,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141145 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:42:40,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00354547 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:42:50,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:50,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:42:50,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149886 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:42:50,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106826 s, 3.8 MB/s\n'; <rc> = 0 >Thread-14540::WARNING::2013-03-17 19:42:53,442::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:42:53,445::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:43:00,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:43:00,985::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:43:00,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:01,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000805825 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:01,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000805825 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:43:01,017::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:43:01,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:43:01,019::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:43:01,020::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:43:01,021::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:43:01,021::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:43:01,022::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:43:01,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:43:01,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:01,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:01,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125305 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:01,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:01,480::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:43:01,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:43:01,483::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:43:01,484::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:43:01,485::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:43:01,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:01,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:01,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:01,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0409648 s, 100 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:43:03,448::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:43:03,450::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:43:03,858::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:43:03,865::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:43:03,867::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:43:11,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:11,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822044 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:11,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:11,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150243 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:43:21,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:21,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100144 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:21,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:21,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800471 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:43:31,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:31,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773171 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:32,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:32,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153021 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:43:41,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:41,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756811 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:42,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:42,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001274 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:43:51,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:43:51,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148359 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:43:52,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:43:52,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838885 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:44:01,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:44:01,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109492 s, 3.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:44:02,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:44:02,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00928907 s, 441 kB/s\n'; <rc> = 0 >Thread-14541::WARNING::2013-03-17 19:44:03,893::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:44:03,897::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:44:11,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:44:11,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784692 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:44:12,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:44:12,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881375 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:44:13,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:44:13,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:44:13,902::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:44:14,367::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:44:14,374::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:44:14,375::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:44:21,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:44:21,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008095 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:44:22,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:44:22,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899966 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:44:31,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:44:31,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823074 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:44:32,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:44:32,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138389 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:44:41,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:44:41,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133595 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:44:42,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:44:42,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135512 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:44:51,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:44:51,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204921 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:44:52,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:44:52,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314492 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:01,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:01,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814631 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:45:02,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:45:02,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171442 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:12,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:12,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105499 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:45:12,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:45:12,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138097 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14542::WARNING::2013-03-17 19:45:14,413::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:45:14,417::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:45:22,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:22,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163865 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:45:22,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:45:22,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855196 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:45:24,420::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:45:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:45:24,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:45:24,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:45:24,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,210::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:45:32,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:45:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:32,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000855159 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:32,244::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000855159 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:45:32,245::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:45:32,245::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:45:32,247::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:45:32,248::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,249::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,250::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,250::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,251::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 19:45:32,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:32,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:32,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:45:32,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:45:32,717::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:45:32,721::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:45:32,723::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:45:32,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104847 s, 391 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:32,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:33,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:33,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182663 s, 224 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:45:42,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:45:42,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015566 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:43,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:43,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112249 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:45:52,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:45:52,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00249563 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:45:53,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:45:53,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936621 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:46:02,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:46:02,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759732 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:46:03,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:46:03,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016526 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:46:12,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:46:12,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136796 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:46:13,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:46:13,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001479 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:46:22,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:46:22,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081946 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:46:23,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:46:23,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119956 s, 341 kB/s\n'; <rc> = 0 >Thread-14543::WARNING::2013-03-17 19:46:24,874::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:46:24,877::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:46:32,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:46:33,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850188 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:46:34,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:46:34,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900398 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:46:34,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:46:34,886::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:46:35,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:46:35,338::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:46:35,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:46:43,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:46:43,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187258 s, 219 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:46:44,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:46:44,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00271399 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:46:53,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:46:53,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791309 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:46:54,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:46:54,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843006 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:47:03,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:47:03,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806388 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:47:04,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:47:04,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141764 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:47:13,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:47:13,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140185 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:47:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:47:14,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905486 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:47:23,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:47:23,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157357 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:47:24,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:47:24,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881079 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:47:33,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:47:33,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808172 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:47:34,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:47:34,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0369561 s, 111 kB/s\n'; <rc> = 0 >Thread-14544::WARNING::2013-03-17 19:47:35,371::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:47:35,377::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:47:43,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:47:43,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147567 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:47:45,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:47:45,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00565226 s, 725 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:47:45,380::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:47:45,382::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:47:45,796::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:47:45,803::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:47:45,804::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:47:53,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:47:53,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085976 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:47:55,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:47:55,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905261 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:03,397::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:48:03,398::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:48:03,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:03,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000686838 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:03,456::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000686838 s, 3.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:48:03,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:48:03,457::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:48:03,459::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:48:03,460::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:48:03,460::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:48:03,461::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:48:03,461::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:48:03,462::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:48:03,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:03,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:03,876::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:48:03,877::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:48:03,880::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:48:03,881::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:48:03,881::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:48:03,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:04,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:04,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:04,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179009 s, 229 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:48:05,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:05,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105131 s, 390 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:14,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:14,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156639 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:48:15,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:15,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074363 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:24,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:24,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106883 s, 383 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:48:25,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149802 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:34,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:34,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146744 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:48:35,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:35,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167825 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:48:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:44,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865648 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:48:45,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:45,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746509 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14545::WARNING::2013-03-17 19:48:45,846::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:48:45,856::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:48:54,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:48:54,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015846 s, 2.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:48:55,860::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:48:55,862::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:55,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:48:55,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141165 s, 290 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:48:56,324::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:48:56,330::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:48:56,331::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:49:04,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:49:04,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149216 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:49:05,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:49:06,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821004 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:49:14,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:49:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882357 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:49:16,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:49:16,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159286 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:49:24,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:49:24,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925627 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:49:26,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:49:26,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877519 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:49:34,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:49:34,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766301 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:49:36,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:49:36,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135967 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:49:44,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:49:45,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149942 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:49:46,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:49:46,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160019 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:49:55,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:49:55,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145857 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14546::WARNING::2013-03-17 19:49:56,361::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:49:56,372::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:49:56,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:49:56,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084331 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:50:05,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:50:05,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900297 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:50:06,374::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:50:06,375::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:50:06,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:06,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:06,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135205 s, 303 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:50:06,825::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:50:06,830::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:50:06,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:50:15,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:50:15,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947284 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:16,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:16,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813449 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:50:25,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:50:25,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145459 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:26,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:26,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274041 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:50:35,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:50:35,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295284 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:36,711::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:50:36,712::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:50:36,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:36,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00161669 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:36,760::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00161669 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:50:36,761::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:50:36,761::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:50:36,763::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:50:36,764::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:50:36,765::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:50:36,765::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:50:36,766::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:50:36,767::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:50:36,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:37,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:37,219::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:50:37,220::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:50:37,223::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:50:37,224::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:50:37,224::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:50:37,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:37,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:37,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:37,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667883 s, 613 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:50:45,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:50:45,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138961 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:47,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:47,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829619 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:50:55,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:50:55,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790574 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:50:57,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:50:57,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837093 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:51:05,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:51:05,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107019 s, 3.8 MB/s\n'; <rc> = 0 >Thread-14547::WARNING::2013-03-17 19:51:06,856::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:51:06,860::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:51:07,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:51:07,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935548 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:51:15,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:51:15,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774352 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:51:16,862::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:51:16,864::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:51:17,282::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:51:17,289::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:51:17,290::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:51:17,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:51:17,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182499 s, 224 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:51:25,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:51:25,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144168 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:51:28,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:51:28,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167119 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:51:35,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:51:35,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963447 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:51:38,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:51:38,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727574 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:51:45,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:51:45,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126149 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:51:48,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:51:48,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156721 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:51:55,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:51:55,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795716 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:51:58,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:51:58,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132745 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:52:05,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:52:05,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152768 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:52:08,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:52:08,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166369 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:52:15,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:52:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164834 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14548::WARNING::2013-03-17 19:52:17,318::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:52:17,324::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:52:18,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:52:18,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872231 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:52:25,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:52:25,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086271 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:52:27,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:52:27,329::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:52:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:52:27,752::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:52:27,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:52:28,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:52:28,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138388 s, 296 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:52:35,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:52:35,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945696 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:52:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:52:38,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089768 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:52:45,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:52:45,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159311 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:52:48,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:52:48,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100735 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:52:55,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:52:55,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942989 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:52:58,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:52:58,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842526 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:05,877::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:53:05,878::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:53:05,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:05,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138699 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:05,924::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138699 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:53:05,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:53:05,926::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:53:05,927::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:53:05,928::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:53:05,929::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:53:05,929::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:53:05,930::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:53:05,930::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:53:05,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:06,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:06,372::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:53:06,373::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 19:53:06,375::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:53:06,376::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:53:06,377::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:53:06,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:06,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:06,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:06,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201359 s, 203 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:53:08,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:53:08,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187102 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:16,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:16,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151338 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:53:18,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:53:18,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167565 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:26,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160021 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14549::WARNING::2013-03-17 19:53:27,781::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:53:27,785::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:53:29,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:53:29,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158655 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:37,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:37,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827374 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:53:37,788::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:53:37,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:53:38,201::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:53:38,208::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:53:38,209::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:53:39,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:53:39,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145777 s, 281 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:47,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:47,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081259 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:53:49,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:53:49,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802057 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:53:57,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:53:57,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166027 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:53:59,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:53:59,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783162 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:54:07,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:54:07,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879542 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:54:09,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:54:09,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104868 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:54:17,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:54:17,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:54:19,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:54:19,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075331 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:54:27,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:54:27,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000657133 s, 6.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:54:29,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:54:29,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974638 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:54:37,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:54:37,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185849 s, 2.2 MB/s\n'; <rc> = 0 >Thread-14550::WARNING::2013-03-17 19:54:38,236::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:54:38,240::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:54:39,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:54:39,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180557 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:54:47,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:54:47,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146553 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:54:48,249::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:54:48,250::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:54:48,673::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:54:48,680::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:54:48,681::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:54:49,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:54:49,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135953 s, 301 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:54:57,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:54:57,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010042 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:54:59,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:54:59,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103043 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:55:07,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:55:07,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130595 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:09,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:09,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155934 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:55:17,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:55:17,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898706 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:19,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:19,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192732 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:55:27,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:55:27,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769129 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:29,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:30,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146232 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:55:37,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:55:37,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752804 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:40,076::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 19:55:40,081::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:55:40,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:40,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00106768 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:40,115::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00106768 s, 1.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 19:55:40,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 19:55:40,116::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 19:55:40,117::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 19:55:40,118::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:55:40,119::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:55:40,119::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:55:40,120::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:55:40,120::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 19:55:40,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:40,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:40,556::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:55:40,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 19:55:40,562::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 19:55:40,563::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 19:55:40,564::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 19:55:40,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:40,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:55:40,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:41,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083948 s, 488 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:55:47,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:55:47,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0345859 s, 118 kB/s\n'; <rc> = 0 >Thread-14551::WARNING::2013-03-17 19:55:48,708::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:55:48,711::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:55:51,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:55:51,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744089 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:55:58,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:55:58,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858443 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:55:58,741::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 19:55:58,743::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:55:58,746::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:55:59,180::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:55:59,187::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:55:59,188::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:56:01,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:56:01,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137252 s, 298 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:56:08,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:56:08,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969492 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:56:11,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:56:11,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839709 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:56:18,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:56:18,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915625 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:56:21,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:56:21,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869469 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:56:28,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:56:28,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151637 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:56:31,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:56:31,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148055 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:56:38,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:56:38,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817607 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:56:41,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:56:41,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780761 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:56:48,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:56:48,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148129 s, 277 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:56:51,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:56:51,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148875 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:56:58,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:56:58,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217665 s, 188 kB/s\n'; <rc> = 0 >Thread-14552::WARNING::2013-03-17 19:56:59,215::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:56:59,219::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 19:57:01,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:57:01,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00959646 s, 427 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:57:08,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:57:08,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896492 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:57:09,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:57:09,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 19:57:09,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:57:09,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:57:09,636::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:57:11,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:57:11,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146003 s, 281 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:57:18,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:57:18,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816828 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:57:21,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:57:21,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959572 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:57:28,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:57:28,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973217 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:57:31,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:57:31,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015209 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:57:38,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:57:38,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750201 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:57:41,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:57:41,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115653 s, 354 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:57:48,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:57:48,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128644 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:57:52,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:57:52,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149274 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:57:58,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:57:58,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120842 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:58:02,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:58:02,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168009 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:08,747::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 19:58:08,748::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:58:08,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:08,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0231339 s, 88.5 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:08,988::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0231339 s, 88.5 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 19:58:08,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 19:58:08,989::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 19:58:08,991::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 19:58:08,992::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:58:09,143::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:58:09,148::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:58:09,149::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 19:58:09,151::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 19:58:09,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:09,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:09,672::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-14553::WARNING::2013-03-17 19:58:09,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-22::DEBUG::2013-03-17 19:58:09,681::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-21::ERROR::2013-03-17 19:58:09,688::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::WARNING::2013-03-17 19:58:09,691::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 19:58:09,692::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 19:58:09,692::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 19:58:09,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:10,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:10,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:10,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868591 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:58:12,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:58:12,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148052 s, 277 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:58:19,690::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:58:19,691::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:20,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 19:58:20,161::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-22::DEBUG::2013-03-17 19:58:20,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019039 s, 215 kB/s\n'; <rc> = 0 >Thread-21::WARNING::2013-03-17 19:58:20,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:58:20,200::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:58:22,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:58:22,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0317796 s, 129 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:30,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:30,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889653 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:58:32,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:58:32,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916946 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:40,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:40,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919435 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:58:42,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:58:42,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993888 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:58:50,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:58:50,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898295 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:58:52,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:58:52,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851702 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:59:00,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:00,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155057 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:59:02,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:59:02,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018856 s, 217 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:59:10,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:10,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764232 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:59:12,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:59:12,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147009 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14554::WARNING::2013-03-17 19:59:20,224::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 19:59:20,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 19:59:20,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:20,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860704 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:59:22,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:59:22,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164965 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:59:30,239::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 19:59:30,240::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:30,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:30,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011668 s, 351 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 19:59:30,682::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 19:59:30,688::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 19:59:30,689::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 19:59:32,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:59:32,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122864 s, 333 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:59:40,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:40,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927874 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:59:42,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:59:42,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827511 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 19:59:50,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 19:59:50,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137273 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 19:59:53,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 19:59:53,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842296 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:00:00,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:00:00,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751921 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:03,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:03,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977127 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:00:10,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:00:10,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847843 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:13,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:13,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159426 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:00:20,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:00:20,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128872 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:23,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:23,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155358 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:00:30,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-14555::WARNING::2013-03-17 20:00:30,732::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:00:30,738::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:00:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222012 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:33,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:33,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831041 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:00:40,741::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:00:40,743::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 20:00:40,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:00:40,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130598 s, 314 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:00:41,217::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:00:41,223::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:00:41,224::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,386::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:00:43,389::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:00:43,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:43,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0132718 s, 154 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:43,438::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0132718 s, 154 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:00:43,438::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:00:43,439::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:00:43,441::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:00:43,442::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,442::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,443::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,443::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,443::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:43,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:43,849::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:00:43,850::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:00:43,853::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:00:43,854::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:00:43,854::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:00:43,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:44,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:44,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:44,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00800882 s, 511 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:00:50,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:00:50,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112435 s, 364 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:00:54,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:00:54,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925874 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:01:00,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:01:00,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874501 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:01:04,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:01:04,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146352 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:01:10,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:01:10,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813329 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:01:14,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:01:14,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785237 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:01:20,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:01:20,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796758 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:01:24,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:01:24,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170233 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:01:30,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:01:31,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147949 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:01:34,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:01:34,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802447 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:01:41,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:01:41,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812741 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14556::WARNING::2013-03-17 20:01:41,258::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:01:41,264::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:01:44,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:01:44,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105927 s, 387 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:01:51,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:01:51,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152579 s, 268 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:01:51,279::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:01:51,280::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:01:51,282::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:01:51,674::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:01:51,680::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:01:51,682::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:01:54,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:01:54,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0466576 s, 87.8 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:02:01,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:02:01,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741749 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:02:04,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:02:04,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124814 s, 328 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:02:11,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:02:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148088 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:02:14,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:02:14,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786668 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:02:21,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:02:21,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015685 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:02:25,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:02:25,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151829 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:02:31,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:02:31,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766314 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:02:35,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:02:35,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823138 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:02:41,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:02:41,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0020764 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:02:45,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:02:45,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172663 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:02:51,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:02:51,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812567 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14557::WARNING::2013-03-17 20:02:51,712::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:02:51,719::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:02:55,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:02:55,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153547 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:01,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:01,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194555 s, 211 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:03:01,722::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:03:01,724::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:03:02,204::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:03:02,225::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:03:02,226::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:03:05,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:03:05,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177906 s, 230 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:11,593::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:03:11,595::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:03:11,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:11,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000836776 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:11,634::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000836776 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:03:11,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:03:11,635::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:03:11,637::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:03:11,638::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:03:11,639::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:03:11,640::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:03:11,640::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:03:11,641::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:03:11,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:12,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:12,057::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:03:12,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:03:12,060::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:03:12,061::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:03:12,062::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:03:12,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:12,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:12,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:12,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010277 s, 399 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:03:15,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:03:15,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210915 s, 194 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:22,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:22,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154083 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:03:25,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:03:25,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830875 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:32,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:32,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148342 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:03:35,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:03:35,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165662 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:42,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:42,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126462 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:03:45,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:03:45,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800518 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:03:52,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:03:52,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138447 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:03:55,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:03:55,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091807 s, 4.5 MB/s\n'; <rc> = 0 >Thread-14558::WARNING::2013-03-17 20:04:02,256::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:04:02,260::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:04:02,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:04:02,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781317 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:04:05,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:04:05,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869052 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:04:12,262::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:04:12,264::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:04:12,691::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-22::DEBUG::2013-03-17 20:04:12,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::WARNING::2013-03-17 20:04:12,745::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:04:12,761::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:04:12,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180312 s, 227 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:04:15,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:04:15,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162515 s, 252 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:04:22,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:04:22,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122093 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:04:25,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:04:26,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162123 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:04:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:04:32,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074483 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:04:36,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:04:36,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960834 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:04:42,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:04:42,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170245 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:04:46,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:04:46,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146759 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:04:52,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:04:53,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155508 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:04:56,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:04:56,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150639 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:05:03,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:03,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139373 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:06,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:06,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152836 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14559::WARNING::2013-03-17 20:05:12,819::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:05:12,823::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:05:13,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:13,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776159 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:16,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:16,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993727 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:05:22,825::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:05:22,826::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:23,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:23,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178782 s, 229 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:05:23,321::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:05:23,327::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:05:23,328::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:26,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:26,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172744 s, 237 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:05:33,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:33,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860653 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:36,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:36,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871873 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:05:43,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:43,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901863 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:46,617::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:05:46,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:05:46,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:46,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00163589 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:46,659::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00163589 s, 1.3 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:05:46,660::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:05:46,660::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:05:46,661::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:05:46,662::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:46,663::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:46,664::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:46,664::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:46,664::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:46,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:47,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:47,110::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:05:47,111::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:05:47,114::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:05:47,115::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:05:47,115::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:05:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:47,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:47,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:47,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0034381 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:05:53,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:05:53,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181355 s, 226 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:05:57,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:05:57,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806639 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:06:03,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:03,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193661 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:06:07,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:06:07,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00671928 s, 610 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:06:13,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:13,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145197 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:06:17,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:06:17,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013812 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14560::WARNING::2013-03-17 20:06:23,358::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:06:23,360::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:06:23,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:23,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286778 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:06:27,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:06:27,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754834 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:06:33,366::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:06:33,368::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:33,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:33,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014935 s, 274 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:06:33,813::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:06:33,819::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:06:33,820::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:06:37,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:06:37,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113381 s, 361 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:06:43,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:43,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817507 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:06:47,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:06:47,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704152 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:06:53,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:06:53,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170072 s, 241 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:06:58,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:06:58,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017582 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:07:03,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:07:03,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725942 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:07:08,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:07:08,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107791 s, 380 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:07:13,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:07:13,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139892 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:07:18,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:07:18,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195254 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:07:23,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:07:23,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015295 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:07:28,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:07:28,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157706 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:07:33,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:07:33,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759391 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14561::WARNING::2013-03-17 20:07:33,846::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:07:33,850::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:07:38,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:07:38,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826681 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:07:43,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 20:07:43,852::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:07:43,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:07:43,856::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 20:07:43,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169664 s, 241 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:07:44,333::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:07:44,340::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:07:44,341::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:07:48,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:07:48,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108197 s, 379 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:07:53,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:07:53,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082621 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:07:58,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:07:58,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772421 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:03,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:03,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080342 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:08:08,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:08:08,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762923 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:13,959::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:08:13,961::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:08:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:14,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131896 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:14,005::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131896 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:08:14,005::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:08:14,005::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:08:14,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:08:14,008::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:14,009::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:14,009::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:14,010::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:14,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:14,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:14,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:14,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:14,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:08:14,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:08:14,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:08:14,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:08:14,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:14,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:14,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:14,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135831 s, 302 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:08:18,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:08:18,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00513997 s, 797 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:24,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145517 s, 281 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:08:28,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:08:28,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000686431 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:08:34,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:34,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170961 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:08:38,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:08:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141512 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14562::WARNING::2013-03-17 20:08:44,368::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:08:44,372::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:08:45,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:45,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00809113 s, 506 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:08:48,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:08:48,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794316 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:08:54,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:08:54,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:08:54,839::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:08:54,846::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:08:54,847::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:08:55,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:08:55,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148909 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:08:58,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:08:59,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00661713 s, 619 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:09:05,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:09:05,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851851 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:09:09,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:09:09,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081916 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:09:15,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:09:15,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143106 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:09:19,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:09:19,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101303 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:09:25,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:09:25,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780101 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:09:29,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:09:29,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177479 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:09:35,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:09:35,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843587 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:09:39,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:09:39,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140841 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:09:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:09:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00811687 s, 505 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:09:49,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:09:49,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014696 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14563::WARNING::2013-03-17 20:09:54,870::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:09:54,874::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:09:55,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:09:55,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0374831 s, 109 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:09:59,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:09:59,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778525 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:10:04,876::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:10:04,878::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:10:05,336::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:10:05,342::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:10:05,344::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:10:05,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:10:05,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019909 s, 206 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:09,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:09,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127691 s, 321 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:10:15,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:10:15,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795819 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:19,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:19,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687026 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:10:25,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:10:25,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862415 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:29,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:29,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831983 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:10:35,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:10:35,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128911 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:39,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:39,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146319 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:10:45,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:10:45,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153537 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:49,808::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:10:49,809::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:10:49,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:49,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000672139 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:49,841::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000672139 s, 3.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:10:49,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:10:49,841::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:10:49,843::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:10:49,844::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:10:49,845::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:10:49,845::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:10:49,846::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:10:49,846::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:10:49,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:50,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:50,297::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:10:50,298::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:10:50,300::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:10:50,301::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:10:50,302::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:10:50,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:50,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:10:50,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:10:50,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185144 s, 221 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:10:55,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:10:55,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173729 s, 236 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:11:00,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:11:00,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747205 s, 548 kB/s\n'; <rc> = 0 >Thread-14564::WARNING::2013-03-17 20:11:05,406::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:11:05,410::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:11:05,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:05,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808095 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:11:10,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:11:10,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149309 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:11:15,412::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:11:15,413::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:15,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0350785 s, 117 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:11:15,916::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:11:15,921::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:11:15,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:11:20,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:11:21,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209687 s, 195 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:11:25,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:25,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858688 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:11:31,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:11:31,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847048 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:11:35,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:35,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150059 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:11:41,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:11:41,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757697 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:11:45,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:45,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105472 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:11:51,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:11:51,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000680306 s, 6.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:11:55,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:11:55,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205296 s, 200 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:12:01,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:12:01,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169197 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:12:06,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:06,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00556778 s, 736 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:12:11,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:12:11,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172833 s, 237 kB/s\n'; <rc> = 0 >Thread-14565::WARNING::2013-03-17 20:12:15,950::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:12:15,954::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:12:16,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:16,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013437 s, 305 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:12:21,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:12:21,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725042 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:12:25,956::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:12:25,957::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:26,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:26,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011741 s, 349 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:12:26,391::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:12:26,397::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:12:26,398::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:12:31,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:12:31,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151391 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:12:36,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:36,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794418 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:12:41,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:12:41,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865152 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:12:46,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586671 s, 698 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:12:51,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:12:51,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881776 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:12:56,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:12:56,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912563 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:13:01,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:13:01,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802857 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:06,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:06,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988017 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:13:11,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:13:11,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179003 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:16,499::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:13:16,501::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:13:16,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:16,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00097222 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:16,544::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00097222 s, 2.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:13:16,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:13:16,545::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:13:16,547::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:13:16,548::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:16,549::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:16,549::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:16,549::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:16,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:16,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:16,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:16,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:16,990::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:13:16,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:13:16,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:13:16,994::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:13:16,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:17,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:17,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:17,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126059 s, 325 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:13:21,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:13:21,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198509 s, 206 kB/s\n'; <rc> = 0 >Thread-14566::WARNING::2013-03-17 20:13:26,423::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:13:26,427::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:13:27,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:27,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723737 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:13:31,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:13:32,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887666 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:13:36,428::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:13:36,430::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:13:36,431::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:13:36,843::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:13:36,849::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:13:36,851::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:13:37,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:37,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135079 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:13:42,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:13:42,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978456 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:47,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:47,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138933 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:13:52,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:13:52,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00511756 s, 800 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:13:57,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:13:57,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089961 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:14:02,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:14:02,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815964 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:14:07,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:14:07,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126918 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:14:12,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:14:12,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144894 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:14:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:14:17,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118068 s, 347 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:14:22,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:14:22,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139342 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:14:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:14:27,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186085 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:14:32,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:14:32,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783695 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14567::WARNING::2013-03-17 20:14:36,878::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:14:36,883::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:14:37,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:14:37,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00401906 s, 1.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:14:42,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:14:42,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884258 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:14:46,886::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:14:46,887::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:14:47,294::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:14:47,301::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:14:47,302::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:14:47,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:14:47,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131992 s, 310 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:14:52,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:14:52,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901162 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:14:57,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:14:57,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125268 s, 327 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:02,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:02,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848698 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:15:07,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:15:08,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000669498 s, 6.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:12,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:12,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144825 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:15:18,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:15:18,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181036 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:22,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:22,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725342 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:15:28,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:15:28,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783739 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:32,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:32,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143263 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:15:38,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:15:38,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116891 s, 350 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:42,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:42,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019425 s, 2.1 MB/s\n'; <rc> = 0 >Thread-14568::WARNING::2013-03-17 20:15:47,329::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:15:47,333::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:15:48,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:15:48,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788556 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:53,033::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:15:53,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:15:53,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:53,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135441 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:53,082::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135441 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:15:53,082::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:15:53,082::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:15:53,084::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:15:53,085::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:15:53,086::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:15:53,086::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:15:53,086::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:15:53,087::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:15:53,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:53,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:53,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:15:53,528::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:15:53,531::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:15:53,532::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:15:53,532::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:15:53,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:53,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:15:53,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:15:53,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00535789 s, 764 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:15:57,336::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:15:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:15:57,765::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:15:57,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:15:57,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:15:58,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:15:58,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150406 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:16:03,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:16:04,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751557 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:16:08,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:16:08,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930753 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:16:14,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:16:14,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198219 s, 207 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:16:18,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:16:18,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804615 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:16:24,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:16:24,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224092 s, 183 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:16:28,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:16:28,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774311 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:16:34,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:16:34,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142489 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:16:38,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:16:38,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142502 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:16:44,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:16:44,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149692 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:16:48,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:16:48,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163436 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:16:54,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:16:54,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092087 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14569::WARNING::2013-03-17 20:16:57,801::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:16:57,807::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:16:58,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:16:58,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870872 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:17:04,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:17:04,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717921 s, 5.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:17:07,809::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:17:07,811::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:17:08,237::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:17:08,244::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:17:08,245::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:17:08,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:17:08,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111378 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:17:14,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:17:14,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143672 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:17:18,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:17:18,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192028 s, 213 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:17:24,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:17:24,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.031793 s, 129 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:17:28,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:17:28,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860165 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:17:34,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:17:34,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818655 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:17:38,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:17:38,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126422 s, 324 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:17:44,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:17:44,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723193 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:17:48,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:17:49,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143516 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:17:54,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:17:54,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0086245 s, 475 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:17:59,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:17:59,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142297 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:18:04,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:18:04,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116941 s, 3.5 MB/s\n'; <rc> = 0 >Thread-14570::WARNING::2013-03-17 20:18:08,269::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:18:08,273::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:18:09,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:09,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804055 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:18:15,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:18:15,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103116 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:18:18,275::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:18:18,276::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:18:18,688::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:18:18,695::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:18:18,697::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,112::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:18:19,115::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:18:19,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:19,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0121519 s, 169 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:18:19,156::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0121519 s, 169 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:18:19,156::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:18:19,157::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:18:19,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:18:19,160::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,161::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,162::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,162::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:19,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:18:19,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:18:19,531::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:18:19,533::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:18:19,535::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:18:19,535::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:18:19,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:19,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:18:19,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:19,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0066265 s, 618 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:18:25,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:18:25,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744654 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:18:29,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:30,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143011 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:18:35,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:18:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0310798 s, 132 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:18:40,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:40,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828954 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:18:45,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:18:45,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113319 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:18:50,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:18:50,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318956 s, 128 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:18:55,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:18:55,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143417 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:19:00,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:19:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872458 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:19:05,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:19:05,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125514 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:19:10,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:19:10,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839815 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:19:15,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:19:15,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224676 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14571::WARNING::2013-03-17 20:19:18,724::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:19:18,727::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:19:20,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:19:20,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108098 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:19:25,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:19:25,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101373 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:19:28,729::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:19:28,730::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:19:28,732::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:19:29,136::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:19:29,152::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:19:29,153::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:19:30,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:19:30,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154815 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:19:35,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:19:35,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016115 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:19:40,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:19:40,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840408 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:19:45,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:19:45,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870109 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:19:50,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:19:50,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015312 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:19:55,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:19:55,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869168 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:20:00,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:20:00,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215382 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:05,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:05,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155383 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:20:10,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:20:10,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150384 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:16,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:16,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145529 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:20:20,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:20:20,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147289 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:26,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163005 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14572::WARNING::2013-03-17 20:20:29,191::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:20:29,198::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:20:30,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:20:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117026 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:36,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:36,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788809 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:20:39,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:20:39,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:20:39,633::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:20:39,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:20:39,641::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:20:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:20:40,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802098 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:46,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:46,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899768 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:20:50,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:20:50,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140126 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:56,307::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:20:56,308::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:20:56,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:56,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000765822 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:56,335::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000765822 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:20:56,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:20:56,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:20:56,339::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:20:56,340::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:20:56,341::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:20:56,342::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:20:56,342::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:20:56,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:20:56,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:56,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:56,775::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:20:56,776::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:20:56,778::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:20:56,780::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:20:56,780::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:20:56,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:57,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:20:57,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:20:57,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923716 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:21:00,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:21:00,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120123 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:21:07,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:21:07,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847306 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:21:10,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:21:10,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146326 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:21:17,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:21:17,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854604 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:21:20,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:21:20,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148116 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:21:27,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:21:27,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179694 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:21:30,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:21:30,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130323 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:21:37,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:21:37,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700923 s, 5.8 MB/s\n'; <rc> = 0 >Thread-14573::WARNING::2013-03-17 20:21:39,666::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:21:39,670::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:21:40,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:21:40,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267724 s, 153 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:21:47,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:21:47,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100292 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:21:49,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:21:49,673::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:21:50,104::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:21:50,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:21:50,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:21:50,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:21:50,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771796 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:21:57,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:21:57,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756608 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:22:00,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:22:00,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793928 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:22:07,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:22:07,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152076 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:22:10,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:22:10,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695042 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:22:17,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:22:17,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014379 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:22:20,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:22:20,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155832 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:22:27,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:22:27,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100363 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:22:31,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:22:31,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110791 s, 370 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:22:37,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:22:37,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160792 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:22:41,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:22:41,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860914 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:22:47,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:22:48,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763744 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14574::WARNING::2013-03-17 20:22:50,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:22:50,153::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:22:51,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:22:51,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00473331 s, 865 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:22:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:22:58,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170001 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:23:00,156::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:23:00,157::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:23:00,558::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:23:00,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:23:00,566::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:01,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:01,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109205 s, 375 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:23:08,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:23:08,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149421 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:11,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:11,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120332 s, 340 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:23:18,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:23:18,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951843 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:21,239::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:23:21,240::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:23:21,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:21,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000735511 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:21,269::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000735511 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:23:21,270::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:23:21,270::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:23:21,272::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:23:21,273::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:21,273::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:21,274::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:21,274::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:21,275::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:21,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:21,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:21,713::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:23:21,714::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:23:21,717::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:23:21,718::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:23:21,718::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:23:21,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:22,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:22,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:22,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169738 s, 241 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:23:28,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:23:28,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740211 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:32,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:32,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124048 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:23:38,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:23:38,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147592 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:42,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:42,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768458 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:23:48,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:23:48,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173496 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:23:52,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:23:52,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143059 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:23:58,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:23:58,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122115 s, 3.4 MB/s\n'; <rc> = 0 >Thread-14575::WARNING::2013-03-17 20:24:00,597::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:24:00,602::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:24:02,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:24:02,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165855 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:24:08,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:24:08,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899407 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:24:10,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:24:10,606::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:24:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:24:11,029::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:24:11,031::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:24:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:24:12,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795637 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:24:18,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:24:18,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897495 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:24:22,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:24:22,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196181 s, 209 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:24:28,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:24:28,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952221 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:24:32,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:24:32,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843764 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:24:38,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:24:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194338 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:24:42,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:24:42,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019854 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:24:48,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:24:48,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998412 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:24:52,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:24:52,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911609 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:24:58,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:24:59,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140074 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:25:02,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:25:02,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864328 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:09,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:09,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110454 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14576::WARNING::2013-03-17 20:25:11,071::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:25:11,075::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:25:12,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:25:12,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152996 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:19,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:19,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800685 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:25:21,077::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:25:21,078::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:25:21,080::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:25:21,476::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:25:21,483::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:25:21,484::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:25:22,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:25:22,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101152 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:29,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:29,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078195 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:25:32,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:25:32,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821575 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:39,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:39,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143596 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:25:42,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:25:42,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230144 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:49,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:49,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148686 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:25:52,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:25:52,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141425 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:59,461::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:25:59,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:25:59,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:59,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000768004 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:59,494::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000768004 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:25:59,494::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:25:59,494::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:25:59,496::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:25:59,501::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:25:59,502::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:25:59,502::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:25:59,502::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:25:59,503::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:25:59,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:25:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:25:59,928::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:25:59,929::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:25:59,933::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:25:59,934::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:25:59,935::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:25:59,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:00,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:26:00,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:00,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121184 s, 338 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:26:02,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:26:02,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569931 s, 719 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:26:10,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:10,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154838 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:26:12,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:26:13,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00620273 s, 660 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:26:20,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:20,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721387 s, 5.7 MB/s\n'; <rc> = 0 >Thread-14577::WARNING::2013-03-17 20:26:21,516::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:26:21,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:26:23,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:26:23,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100776 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:26:30,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:30,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842037 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:26:31,525::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:26:31,527::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:26:31,934::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:26:31,941::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:26:31,942::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:26:33,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:26:33,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170418 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:26:40,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:40,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136008 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:26:43,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:26:43,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150639 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:26:50,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:26:50,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116463 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:26:53,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:26:53,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122645 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:27:00,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:27:00,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008262 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:27:03,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:27:03,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140956 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:27:10,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:27:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916682 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:27:13,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:27:13,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00343566 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:27:21,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:27:21,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077858 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:27:23,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:27:23,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171257 s, 239 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:27:31,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:27:31,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083715 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14578::WARNING::2013-03-17 20:27:31,977::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:27:31,981::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:27:33,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:27:33,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118327 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:27:41,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:27:41,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836501 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:27:41,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:27:41,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:27:42,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:27:42,434::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:27:42,435::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:27:43,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:27:43,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149059 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:27:51,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:27:51,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888403 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:27:53,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:27:53,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146578 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:28:01,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:28:01,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127576 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:03,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158362 s, 259 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:28:11,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:28:11,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802994 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:13,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:13,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232522 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:28:21,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:28:21,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127883 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:23,838::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:28:23,839::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:28:23,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:23,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128739 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:23,879::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128739 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:28:23,880::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:28:23,880::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:28:23,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:28:23,882::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:23,883::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:23,883::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:23,884::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:23,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:23,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:24,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:24,337::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:24,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:28:24,340::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:28:24,341::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:28:24,342::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:28:24,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:24,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:24,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:24,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162342 s, 252 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:28:31,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:28:31,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963553 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:28:34,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:34,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147267 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:28:41,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:28:42,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157253 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14579::WARNING::2013-03-17 20:28:42,473::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:28:42,483::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:28:44,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:44,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144993 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:28:52,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:28:52,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127903 s, 320 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:28:52,488::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:28:52,489::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:28:52,949::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:28:52,956::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:28:52,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:28:54,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:28:54,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778687 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:29:02,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:29:02,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838339 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:29:04,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:29:04,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929819 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:29:12,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:29:12,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075884 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:29:14,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:29:15,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798994 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:29:22,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:29:22,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813731 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:29:25,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:29:25,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018754 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:29:32,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:29:32,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765421 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:29:35,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:29:35,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882536 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:29:42,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:29:42,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141082 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:29:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:29:45,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147607 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:29:52,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:29:52,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906153 s, 4.5 MB/s\n'; <rc> = 0 >Thread-14580::WARNING::2013-03-17 20:29:52,987::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:29:52,990::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:29:55,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:29:55,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014929 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:30:02,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 20:30:02,996::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:30:03,001::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 20:30:03,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103898 s, 394 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:30:03,491::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:30:03,496::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:30:03,498::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:30:05,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:30:05,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201901 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:30:13,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:30:13,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935819 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:30:15,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:30:15,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870076 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:30:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:30:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718397 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:30:25,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:30:25,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153626 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:30:33,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:30:33,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146169 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:30:35,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:30:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090092 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:30:43,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:30:43,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749751 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:30:45,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:30:45,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156451 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:30:53,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:30:53,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760805 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:30:55,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:30:55,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962829 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:03,458::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:31:03,460::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:31:03,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:03,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00068538 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14581::WARNING::2013-03-17 20:31:03,528::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-20::DEBUG::2013-03-17 20:31:03,534::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00068538 s, 3.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:31:03,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:31:03,540::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-21::ERROR::2013-03-17 20:31:03,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:31:03,558::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:31:03,559::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:03,560::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:03,560::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:03,561::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:03,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:03,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:03,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:03,995::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:03,996::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:31:03,999::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:31:04,000::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:31:04,001::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:31:04,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:04,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:04,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:04,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00332144 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:31:05,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:31:05,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098386 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:31:13,558::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:31:13,559::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:31:13,561::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:31:14,000::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:31:14,009::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:31:14,010::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:31:14,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:14,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153082 s, 268 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:31:15,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:31:15,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152308 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:24,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:24,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819168 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:31:25,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:31:25,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00448367 s, 914 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:34,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:34,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785321 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:31:35,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:31:35,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104833 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:44,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:44,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986131 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:31:45,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:31:45,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142162 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:31:54,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:31:54,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012591 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:31:55,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:31:55,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153538 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:32:04,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:32:04,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144552 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:32:05,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:32:05,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135071 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14582::WARNING::2013-03-17 20:32:14,047::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:32:14,050::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:32:14,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:32:14,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851038 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:32:15,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:32:15,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00534922 s, 766 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:32:24,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:32:24,054::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:32:24,531::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:32:24,537::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:32:24,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:32:25,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:32:25,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131076 s, 312 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:32:25,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:32:25,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189905 s, 216 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:32:35,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:32:35,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096833 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:32:35,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:32:35,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923912 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:32:45,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:32:45,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014977 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:32:45,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:32:45,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00388119 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:32:55,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:32:55,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920652 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:32:55,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:32:55,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749734 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:33:05,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:33:05,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154164 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:05,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:06,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155521 s, 263 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:33:15,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:33:15,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778443 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:16,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:16,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086005 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14583::WARNING::2013-03-17 20:33:24,568::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:33:24,572::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:33:25,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:33:25,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731866 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:26,088::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:33:26,090::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:33:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:26,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000685303 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:26,117::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000685303 s, 3.0 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:33:26,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:33:26,118::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:33:26,119::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:33:26,120::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:33:26,121::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:33:26,121::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:33:26,122::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:33:26,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:33:26,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:26,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:26,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:33:26,540::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:33:26,542::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:33:26,543::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:33:26,544::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:33:26,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:26,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:26,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:27,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169213 s, 242 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:33:34,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:33:34,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:33:34,999::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:33:35,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:33:35,008::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:33:35,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:33:35,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134042 s, 306 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:37,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:37,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919645 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:33:45,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:33:45,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092169 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:47,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:47,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757163 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:33:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:33:55,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159608 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:33:57,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:33:57,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001437 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:34:05,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:34:05,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756668 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:34:07,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:34:07,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141786 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:34:15,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:34:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175095 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:34:17,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:34:17,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687524 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:34:25,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:34:25,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175587 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:34:27,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:34:27,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151215 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14584::WARNING::2013-03-17 20:34:35,034::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:34:35,038::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:34:36,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:34:36,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149642 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:34:37,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:34:37,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088235 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:34:45,041::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:34:45,043::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:34:45,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:34:45,515::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:34:45,516::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:34:46,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:34:46,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137147 s, 299 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:34:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:34:47,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138899 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:34:56,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:34:56,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773658 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:34:57,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:34:57,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090594 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:35:06,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:35:06,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177383 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:35:07,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:35:07,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209775 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:35:16,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:35:16,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147454 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:35:17,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:35:17,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133673 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:35:26,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:35:26,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139409 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:35:27,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:35:27,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141248 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:35:36,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:35:36,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795513 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:35:37,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:35:37,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132699 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14585::WARNING::2013-03-17 20:35:45,541::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:35:45,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:35:46,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:35:46,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882311 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:35:47,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:35:47,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841034 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:35:55,547::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:35:55,548::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:35:55,999::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:35:56,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:35:56,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:35:56,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:35:56,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116361 s, 352 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:35:57,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:35:57,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769981 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:06,714::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:36:06,716::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:36:06,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:06,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00569969 s, 359 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:06,752::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00569969 s, 359 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:36:06,753::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:36:06,753::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:36:06,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:36:06,756::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:36:06,757::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:36:06,757::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:36:06,758::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:36:06,758::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:36:06,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:07,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:07,196::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:36:07,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:36:07,201::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:36:07,202::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:36:07,203::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:36:07,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:07,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:07,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:07,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00533351 s, 768 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:07,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:36:07,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128094 s, 320 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:17,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:17,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:17,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118427 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:17,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143394 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:27,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:27,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:27,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983556 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:27,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118366 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:37,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:36:38,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000638528 s, 6.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:38,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:38,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985622 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:36:48,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:36:48,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171059 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:48,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:48,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765874 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14586::WARNING::2013-03-17 20:36:56,036::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:36:56,039::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:36:58,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:36:58,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844402 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:36:58,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:36:58,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871291 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:37:06,041::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:37:06,042::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:37:06,043::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:37:06,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:37:06,465::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:37:06,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:37:08,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:37:08,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844478 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:37:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:37:08,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181006 s, 226 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:37:18,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:37:18,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224938 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:37:18,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:37:18,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179675 s, 228 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:37:28,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:37:28,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904692 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:37:28,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:37:28,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790548 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:37:38,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:37:38,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00320262 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:37:38,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:37:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808578 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:37:48,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:37:48,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894631 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:37:48,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:37:48,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789409 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:37:58,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:37:58,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873398 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:37:58,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:37:58,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00794445 s, 516 kB/s\n'; <rc> = 0 >Thread-14587::WARNING::2013-03-17 20:38:06,500::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:38:06,504::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:38:08,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:08,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000685883 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:38:08,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:38:08,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762402 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:38:16,506::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:38:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:38:16,915::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:38:16,922::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:38:16,924::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:18,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:18,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126494 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:38:18,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:38:18,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149928 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:28,424::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:38:28,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:38:28,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:28,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00084704 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:28,456::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00084704 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:38:28,456::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:38:28,456::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:38:28,458::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:38:28,465::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:28,466::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:28,466::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:28,467::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:28,467::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:28,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:28,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:28,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:38:28,891::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-20::DEBUG::2013-03-17 20:38:28,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::WARNING::2013-03-17 20:38:28,902::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:38:28,929::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:38:28,943::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:38:28,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 20:38:28,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215679 s, 190 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:29,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:29,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:29,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157252 s, 260 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:38:39,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:38:39,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152766 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:39,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:39,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768443 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:38:49,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:38:49,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846704 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:49,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:49,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830808 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:38:59,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:38:59,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149224 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:38:59,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:38:59,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787518 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:39:09,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:39:09,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166357 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:39:09,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:39:09,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151693 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14588::WARNING::2013-03-17 20:39:16,950::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:39:16,954::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:39:19,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:39:19,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807625 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:39:19,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:39:19,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00488203 s, 839 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:39:26,956::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:39:26,958::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:39:27,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:39:27,435::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:39:27,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:39:29,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:39:29,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145214 s, 282 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:39:29,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:39:29,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119534 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:39:39,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:39:39,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072943 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:39:39,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:39:39,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101252 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:39:49,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:39:49,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159625 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:39:49,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:39:49,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745366 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:39:59,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:39:59,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110606 s, 370 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:39:59,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:39:59,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100932 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:40:09,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:40:09,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080826 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:40:09,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:40:09,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159877 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:40:19,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:40:19,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122877 s, 333 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:40:19,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:40:20,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855002 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14589::WARNING::2013-03-17 20:40:27,461::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:40:27,465::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 20:40:29,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:40:29,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936922 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:40:30,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:40:30,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016719 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:40:37,467::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:40:37,469::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:40:37,874::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:40:37,881::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:40:37,883::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:40:39,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:40:40,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168585 s, 243 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:40:40,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:40:40,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008477 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:40:50,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:40:50,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884971 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:40:50,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:40:50,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126114 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:41:00,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:00,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:00,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103916 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:41:00,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201093 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:41:10,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:41:10,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717962 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:10,436::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:41:10,437::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:41:10,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:10,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128858 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:10,481::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128858 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:41:10,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:41:10,482::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:41:10,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:41:10,485::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:41:10,485::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:41:10,486::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:41:10,486::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:41:10,487::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:41:10,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:10,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:10,913::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:41:10,914::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:41:10,917::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:41:10,918::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:41:10,918::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:41:10,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:11,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:11,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:11,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013508 s, 303 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:41:20,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:41:20,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806112 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:21,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:21,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159591 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:41:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:41:30,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131913 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:31,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:31,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280657 s, 146 kB/s\n'; <rc> = 0 >Thread-14590::WARNING::2013-03-17 20:41:37,911::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:41:37,917::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:41:40,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:41:40,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00068649 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:41,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:41,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103522 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:41:47,919::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:41:47,926::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:41:48,346::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:41:48,353::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:41:48,354::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:41:50,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:41:50,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803257 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:41:51,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:41:51,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00704182 s, 582 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:42:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:42:00,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863237 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:42:01,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:42:01,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155541 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:42:10,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:42:10,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142306 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:42:11,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:42:11,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077377 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:42:20,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:42:20,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140905 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:42:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:42:21,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137602 s, 298 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:42:30,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:42:30,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933446 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:42:31,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:42:32,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150745 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:42:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:42:40,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808916 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:42:42,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:42:42,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182037 s, 2.3 MB/s\n'; <rc> = 0 >Thread-14591::WARNING::2013-03-17 20:42:48,383::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:42:48,389::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:42:50,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:42:50,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141123 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:42:52,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:42:52,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807826 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:42:58,391::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:42:58,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:42:58,393::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:42:58,797::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:42:58,803::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:42:58,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:00,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:00,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101471 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:43:02,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:43:02,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018014 s, 227 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:10,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:10,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100285 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:43:12,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:43:12,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143922 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:20,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:20,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805019 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:43:22,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:43:22,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204916 s, 200 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:30,973::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:43:30,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:43:30,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:31,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138945 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:31,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138945 s, 1.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:43:31,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:43:31,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:43:31,019::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:43:31,020::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:31,021::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:31,021::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:31,022::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:31,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:31,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:31,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:31,445::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:43:31,446::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:43:31,448::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:43:31,450::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:43:31,450::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:43:31,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:31,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:31,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:31,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182872 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:43:32,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:43:32,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137984 s, 297 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:41,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:41,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016572 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:43:42,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:43:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176143 s, 233 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:43:51,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:43:51,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122805 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:43:52,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:43:52,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158936 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14592::WARNING::2013-03-17 20:43:58,832::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:43:58,843::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:44:01,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:44:02,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140741 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:44:02,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:44:02,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013875 s, 295 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:44:08,847::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:44:08,849::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:44:09,258::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:44:09,265::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:44:09,266::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:44:12,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:44:12,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897949 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:44:12,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:44:12,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021833 s, 188 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:44:22,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:44:22,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818284 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:44:22,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:44:23,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307193 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:44:32,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:44:32,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154286 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:44:33,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:44:33,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157699 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:44:42,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:44:42,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010284 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:44:43,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:44:43,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146892 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:44:52,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:44:52,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155042 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:44:53,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:44:53,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200835 s, 2.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:45:02,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:45:02,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115505 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:45:03,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:45:03,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015116 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14593::WARNING::2013-03-17 20:45:09,296::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:45:09,301::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:45:12,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:45:12,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814522 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:45:13,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:45:13,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916774 s, 4.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:45:19,304::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:45:19,305::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:45:19,736::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:45:19,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:45:19,744::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:45:22,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:45:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015337 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:45:23,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:45:23,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176788 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:45:32,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:45:32,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120749 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:45:33,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:45:33,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872146 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:45:42,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:45:42,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133155 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:45:43,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:45:43,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136379 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:45:52,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:45:52,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796893 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:45:53,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:45:53,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139723 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:46:02,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:46:02,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138522 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:03,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:03,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729771 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:46:12,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:46:12,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134669 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:13,851::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:46:13,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:46:13,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:13,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132021 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:13,897::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132021 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:46:13,897::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:46:13,897::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:46:13,899::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:46:13,900::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:46:13,901::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:46:13,901::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:46:13,902::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:46:13,902::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:46:13,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:14,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:14,355::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:46:14,356::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:46:14,359::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:46:14,360::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:46:14,361::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:46:14,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:14,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:14,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:14,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290163 s, 1.4 MB/s\n'; <rc> = 0 >Thread-14594::WARNING::2013-03-17 20:46:19,771::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:46:19,775::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:46:22,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:46:22,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795225 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:24,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:24,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00628204 s, 652 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:46:29,777::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:46:29,778::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:46:30,184::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:46:30,191::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:46:30,192::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:46:32,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:46:32,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873122 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:34,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:34,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159291 s, 257 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:46:42,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:46:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865929 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:44,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:45,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148072 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:46:52,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:46:52,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146572 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:46:55,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:46:55,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755476 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:47:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:47:02,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136797 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:47:05,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:47:05,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887236 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:47:12,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:47:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125727 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:47:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:47:15,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083798 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:47:22,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:47:22,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165493 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:47:25,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:47:25,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.009755 s, 420 kB/s\n'; <rc> = 0 >Thread-14595::WARNING::2013-03-17 20:47:30,224::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:47:30,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:47:32,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:47:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136596 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:47:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:47:35,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983196 s, 4.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:47:40,234::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:47:40,235::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:47:40,630::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:47:40,637::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:47:40,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:47:42,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:47:42,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140288 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:47:45,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:47:45,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118836 s, 345 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:47:52,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:47:52,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813718 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:47:55,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:47:55,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074783 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:02,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:02,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161641 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:48:05,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:48:05,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225715 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:13,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:13,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715259 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:48:15,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:48:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135315 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:23,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:23,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179409 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:48:25,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:48:25,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264034 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:33,080::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:48:33,084::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:48:33,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:33,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762602 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:33,122::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762602 s, 2.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:48:33,122::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:48:33,122::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:48:33,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:48:33,125::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:33,125::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:33,126::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:33,126::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:33,126::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:33,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:33,540::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:33,541::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:48:33,543::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:48:33,544::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:48:33,545::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:48:33,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:33,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:48:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183867 s, 223 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:48:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:48:35,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0091913 s, 446 kB/s\n'; <rc> = 0 >Thread-14596::WARNING::2013-03-17 20:48:40,666::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:48:40,670::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:48:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:43,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804853 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:48:45,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:48:45,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818667 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:48:50,672::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:48:50,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:48:50,674::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:48:51,080::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:48:51,087::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:48:51,088::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:48:53,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:48:54,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019486 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:48:55,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:48:56,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167634 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:49:04,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:49:04,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148746 s, 275 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:49:06,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:49:06,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886557 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:49:14,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:49:14,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136834 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:49:16,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:49:16,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148304 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:49:24,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:49:24,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112133 s, 365 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:49:26,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:49:26,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014552 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:49:34,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:49:34,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131546 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:49:36,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:49:36,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746529 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:49:44,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:49:44,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163498 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:49:46,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:49:46,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947568 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14597::WARNING::2013-03-17 20:49:51,117::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:49:51,121::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:49:54,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:49:54,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896919 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:49:56,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:49:56,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803405 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:50:01,123::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:50:01,124::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:50:01,804::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:50:01,811::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:50:01,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:50:04,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:50:04,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146283 s, 280 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:50:06,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:50:06,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163027 s, 251 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:50:14,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:50:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849272 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:50:16,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:50:16,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887982 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:50:24,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:50:24,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161306 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:50:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:50:26,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116748 s, 351 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:50:34,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:50:34,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128863 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:50:36,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:50:36,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848915 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:50:44,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:50:44,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104097 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:50:46,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:50:46,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135068 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:50:54,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:50:54,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788025 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:50:56,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:50:57,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221536 s, 1.8 MB/s\n'; <rc> = 0 >Thread-14598::WARNING::2013-03-17 20:51:01,842::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:51:01,849::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:51:04,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:51:04,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013078 s, 313 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:07,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:07,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852088 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:51:11,852::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:51:11,853::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:51:12,306::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:51:12,316::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:51:12,317::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:51:14,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:51:14,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001431 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:17,118::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:51:17,120::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:51:17,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:17,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0186308 s, 110 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:17,169::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0186308 s, 110 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:51:17,170::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:51:17,170::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:51:17,172::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:51:17,173::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:51:17,174::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:51:17,175::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:51:17,175::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:51:17,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:51:17,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:17,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:17,615::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:51:17,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:51:17,618::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:51:17,620::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:51:17,620::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:51:17,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:18,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:18,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:18,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0035134 s, 1.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:51:24,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:51:24,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751927 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:28,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:28,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806675 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:51:34,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:51:34,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730213 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:38,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:38,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170211 s, 241 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:51:44,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:51:44,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810337 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:48,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:48,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831275 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:51:54,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:51:54,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014397 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:51:58,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:51:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158309 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:52:04,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:52:04,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127822 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:52:08,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:52:08,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156441 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14599::WARNING::2013-03-17 20:52:12,347::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:52:12,353::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:52:14,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:52:14,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772636 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:52:18,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:52:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802924 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:52:22,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:52:22,357::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:52:22,763::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:52:22,770::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:52:22,771::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:52:24,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:52:24,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754314 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:52:28,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:52:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00984853 s, 416 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:52:34,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:52:35,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134383 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:52:38,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:52:38,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841144 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:52:45,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:52:45,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161755 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:52:48,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:52:48,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904387 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:52:55,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:52:55,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209114 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:52:58,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:52:58,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910004 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:05,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:05,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716621 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:53:08,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:53:08,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717318 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:15,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154081 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:53:18,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:53:18,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170951 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14600::WARNING::2013-03-17 20:53:22,808::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:53:22,811::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:53:25,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:25,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077498 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:53:29,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:53:29,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920694 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:53:32,814::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:53:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:53:33,228::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:53:33,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:53:33,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,248::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:53:35,251::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:53:35,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:35,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131405 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:35,288::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131405 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:53:35,289::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:53:35,289::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:53:35,291::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:53:35,292::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,292::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,293::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,293::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,293::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:35,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:35,738::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:53:35,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:53:35,741::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:53:35,742::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:53:35,742::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:53:35,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:36,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:36,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:36,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152915 s, 268 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:53:39,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:53:39,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00935516 s, 438 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:46,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:46,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160414 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:53:49,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:53:49,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784267 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:53:56,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:53:56,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162526 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:53:59,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:53:59,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197031 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:54:06,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:54:06,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084241 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:54:09,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:54:09,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014916 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:54:16,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:54:16,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764873 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:54:19,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:54:19,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861779 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:54:26,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:54:26,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147106 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:54:29,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:54:29,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167494 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14601::WARNING::2013-03-17 20:54:33,264::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:54:33,271::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:54:36,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:54:36,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873445 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:54:39,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:54:39,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122043 s, 3.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:54:43,273::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 20:54:43,274::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:54:43,275::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:54:43,683::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:54:43,691::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:54:43,692::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:54:46,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:54:46,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789839 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:54:49,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:54:49,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111286 s, 368 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:54:56,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:54:56,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015257 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:54:59,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:54:59,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162961 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:55:06,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:55:06,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132991 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:55:09,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:55:09,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174221 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:55:16,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:55:16,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110737 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:55:19,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:55:19,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770416 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:55:26,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:55:26,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119118 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:55:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:55:29,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783135 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:55:36,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:55:36,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747197 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:55:40,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:55:40,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937972 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14602::WARNING::2013-03-17 20:55:43,720::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:55:43,723::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:55:46,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:55:46,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807685 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:55:50,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:55:50,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104443 s, 3.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:55:53,728::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:55:53,729::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:55:54,164::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:55:54,171::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:55:54,172::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:55:56,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:55:56,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848751 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:00,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:00,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108907 s, 376 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:56:06,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:56:06,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187738 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:10,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:10,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850645 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:56:16,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:56:16,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00526495 s, 778 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:20,341::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 20:56:20,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:56:20,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:20,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00123797 s, 1.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:20,383::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00123797 s, 1.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 20:56:20,384::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 20:56:20,384::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 20:56:20,385::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 20:56:20,386::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:56:20,387::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:56:20,388::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:56:20,388::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:56:20,388::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 20:56:20,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:20,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:20,843::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 20:56:20,845::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 20:56:20,848::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 20:56:20,849::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 20:56:20,849::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 20:56:20,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:21,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:21,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:21,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163596 s, 250 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:56:26,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:56:26,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849728 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:31,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:31,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014265 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:56:36,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:56:36,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849601 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:41,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:41,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167442 s, 245 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:56:46,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:56:46,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139083 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:56:51,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:56:51,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801023 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14603::WARNING::2013-03-17 20:56:54,201::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:56:54,207::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:56:56,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:56:57,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144147 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:57:01,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:57:01,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826107 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:57:04,209::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:57:04,210::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:57:04,632::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:57:04,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:57:04,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:57:07,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:57:07,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890303 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:57:11,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:57:11,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189935 s, 216 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:57:17,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:57:17,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789102 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:57:21,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:57:21,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089769 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:57:27,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:57:27,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824422 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:57:31,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:57:31,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213347 s, 192 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:57:37,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:57:37,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781818 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:57:41,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:57:42,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108576 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:57:47,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:57:47,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136486 s, 300 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:57:52,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:57:52,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100753 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:57:57,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:57:57,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145392 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:58:02,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:58:02,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935748 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14604::WARNING::2013-03-17 20:58:04,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:58:04,671::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:58:07,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:07,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00836963 s, 489 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:58:12,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:58:12,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153321 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:58:14,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:58:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:58:15,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:58:15,167::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:58:15,168::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:17,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:17,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852181 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:58:22,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:58:22,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178812 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:27,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:27,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777324 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:58:32,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:58:32,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178519 s, 229 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:37,365::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 20:58:37,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:58:37,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:37,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131107 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:37,411::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131107 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 20:58:37,412::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 20:58:37,412::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 20:58:37,414::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 20:58:37,415::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:37,415::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:37,416::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:37,416::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:37,416::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:37,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:37,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:58:37,857::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 20:58:37,859::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 20:58:37,860::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 20:58:37,861::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 20:58:37,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:38,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:38,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:38,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286563 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:58:42,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:58:42,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020955 s, 195 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:48,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:48,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081139 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:58:52,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:58:52,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812873 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:58:58,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:58:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128219 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:59:02,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:59:02,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172158 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:59:08,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:59:08,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00303129 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:59:12,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:59:12,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011337 s, 3.6 MB/s\n'; <rc> = 0 >Thread-14605::WARNING::2013-03-17 20:59:15,205::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 20:59:15,211::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 20:59:18,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:59:18,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163017 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:59:22,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:59:22,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857476 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 20:59:25,214::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 20:59:25,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 20:59:25,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 20:59:25,739::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 20:59:25,741::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 20:59:28,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:59:28,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900052 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:59:32,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:59:32,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177249 s, 231 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:59:38,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:59:38,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144554 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:59:42,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:59:42,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081157 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:59:48,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:59:48,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141035 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 20:59:53,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 20:59:53,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824658 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 20:59:58,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 20:59:58,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000662006 s, 6.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:00:03,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:00:03,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162638 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:00:08,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:00:08,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164679 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:00:13,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:00:13,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107358 s, 382 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:00:18,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:00:18,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111068 s, 3.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:00:23,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:00:23,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979828 s, 4.2 MB/s\n'; <rc> = 0 >Thread-14606::WARNING::2013-03-17 21:00:25,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:00:25,772::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:00:28,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:00:28,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789384 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:00:33,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:00:33,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788188 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:00:35,810::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:00:35,811::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:00:35,812::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:00:36,244::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:00:36,250::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:00:36,252::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:00:38,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:00:38,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010378 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:00:43,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:00:43,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018667 s, 219 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:00:48,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:00:48,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013402 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:00:53,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:00:53,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140098 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:00:58,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:00:58,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184608 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:03,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:03,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207559 s, 197 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:01:08,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:01:08,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135618 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:13,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:13,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740521 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:01:18,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:01:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134845 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:23,738::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:01:23,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:01:23,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:23,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125094 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:23,771::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125094 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:01:23,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:01:23,771::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:01:23,773::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:01:23,774::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:01:23,775::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:01:23,775::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:01:23,776::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:01:23,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:01:23,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:24,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:24,218::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:01:24,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:01:24,223::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:01:24,224::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:01:24,224::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:01:24,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:24,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:24,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:24,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00812297 s, 504 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:01:28,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:01:28,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153723 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:34,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:34,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727013 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14607::WARNING::2013-03-17 21:01:36,280::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:01:36,284::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:01:38,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:01:39,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102026 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:45,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:45,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862569 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:01:46,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:01:46,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:01:46,726::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:01:46,733::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:01:46,734::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:01:49,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:01:49,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901967 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:01:55,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:01:55,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01405 s, 292 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:01:59,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:01:59,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839408 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:02:05,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:02:05,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792063 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:02:09,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:02:09,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195032 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:02:15,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:02:15,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162604 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:02:19,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:02:19,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801029 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:02:25,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:02:25,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849534 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:02:29,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:02:29,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153452 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:02:35,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:02:35,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203712 s, 201 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:02:39,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:02:39,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151709 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:02:45,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:02:45,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774073 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14608::WARNING::2013-03-17 21:02:46,760::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:02:46,764::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:02:49,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:02:49,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787592 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:02:55,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:02:55,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775132 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:02:56,767::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:02:56,768::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:02:57,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:02:57,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:02:57,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:02:59,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:02:59,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851494 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:03:06,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:03:06,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102031 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:09,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129852 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:03:16,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:03:16,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785085 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:19,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:19,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126361 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:03:26,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:03:26,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100597 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:29,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:29,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892192 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:03:36,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:03:36,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780143 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:39,418::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:03:39,420::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:03:39,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:39,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00159393 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:39,468::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00159393 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:03:39,469::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:03:39,469::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:03:39,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:03:39,471::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:03:39,472::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:03:39,473::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:03:39,473::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:03:39,473::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:03:39,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:39,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:39,884::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:03:39,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:03:39,890::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:03:39,893::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:03:39,893::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:03:39,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:40,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:40,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:40,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135232 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:03:46,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:03:46,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135847 s, 302 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:03:50,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:03:50,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793643 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:03:56,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:03:56,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218978 s, 1.9 MB/s\n'; <rc> = 0 >Thread-14609::WARNING::2013-03-17 21:03:57,247::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:03:57,251::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:04:00,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:04:00,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136624 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:04:06,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:04:06,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145841 s, 281 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:04:07,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:04:07,255::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:04:07,683::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:04:07,692::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:04:07,693::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:04:10,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:04:10,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016544 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:04:16,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:04:16,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795549 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:04:20,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:04:20,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855393 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:04:26,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:04:26,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860048 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:04:30,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:04:30,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922806 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:04:36,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:04:36,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819942 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:04:40,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:04:40,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986751 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:04:47,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:04:47,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177899 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:04:50,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:04:50,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139272 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:04:57,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:04:57,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152239 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:05:00,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:05:00,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156362 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:05:07,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:05:07,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00601178 s, 681 kB/s\n'; <rc> = 0 >Thread-14610::WARNING::2013-03-17 21:05:07,722::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:05:07,728::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:05:10,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:05:10,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165396 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:05:17,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:05:17,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146652 s, 279 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:05:17,731::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:05:17,733::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:05:18,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:05:18,154::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:05:18,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:05:20,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:05:20,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087167 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:05:27,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:05:27,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842105 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:05:30,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:05:30,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00415694 s, 985 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:05:37,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:05:37,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788092 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:05:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:05:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139448 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:05:47,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:05:47,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775234 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:05:50,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:05:50,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398727 s, 1.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:05:57,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:05:57,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141369 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:06:00,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:06:00,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154009 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:07,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:07,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756936 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:06:10,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:06:10,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893019 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:17,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:18,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141279 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14611::WARNING::2013-03-17 21:06:18,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:06:18,200::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:06:20,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:06:20,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880819 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:06:28,214::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:06:28,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:06:28,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:28,238::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:06:28,243::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:06:28,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:28,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00854084 s, 240 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:28,305::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00854084 s, 240 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:06:28,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:06:28,305::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:06:28,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:06:28,308::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-21::DEBUG::2013-03-17 21:06:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:06:28,626::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:06:28,626::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:28,658::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:28,666::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:28,667::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:28,668::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:28,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:28,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:29,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:29,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:06:29,063::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:06:29,066::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:06:29,067::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:06:29,067::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:06:29,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:29,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:29,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:29,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138054 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:06:30,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:06:31,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754408 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:39,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:39,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00492226 s, 832 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:06:41,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:06:41,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845664 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:49,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:49,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141104 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:06:51,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:06:51,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757359 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:06:59,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:06:59,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836384 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:07:01,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:07:01,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147414 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:07:09,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:07:09,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771762 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:07:11,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:07:11,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950811 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:07:19,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:07:19,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147214 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:07:21,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:07:21,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104488 s, 3.9 MB/s\n'; <rc> = 0 >Thread-14612::WARNING::2013-03-17 21:07:28,677::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:07:28,683::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:07:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:07:29,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086045 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:07:31,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:07:31,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115171 s, 356 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:07:38,686::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:07:38,688::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:07:39,109::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:07:39,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:07:39,117::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:07:39,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:07:40,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101925 s, 402 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:07:41,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:07:41,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172738 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:07:50,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:07:50,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843432 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:07:51,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:07:51,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834054 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:08:00,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:08:00,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151772 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:01,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:01,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00631553 s, 649 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:08:10,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:08:10,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772661 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:11,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:11,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922653 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:08:20,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:08:20,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157892 s, 259 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:21,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:21,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730201 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:08:30,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:08:30,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134546 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:31,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:31,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794998 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14613::WARNING::2013-03-17 21:08:39,157::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:08:39,162::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:08:40,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:08:40,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151556 s, 270 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:41,737::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:08:41,738::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:08:41,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:41,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000799605 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:41,769::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000799605 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:08:41,770::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:08:41,770::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:08:41,772::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:08:41,773::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:08:41,773::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:08:41,774::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:08:41,774::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:08:41,775::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:08:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:42,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:42,208::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:08:42,209::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:08:42,211::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:08:42,213::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:08:42,213::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:08:42,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:42,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:42,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:42,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00469013 s, 873 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:08:49,165::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:08:49,166::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:08:49,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:08:49,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:08:49,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:08:50,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:08:50,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127049 s, 322 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:08:52,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:08:52,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829065 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:09:00,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:09:00,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110851 s, 370 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:09:02,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:09:02,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787814 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:09:10,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:09:10,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138715 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:09:12,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:09:12,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178153 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:09:20,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:09:20,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956166 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:09:22,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:09:22,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875349 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:09:30,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:09:30,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00469273 s, 873 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:09:32,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:09:32,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704695 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:09:40,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:09:41,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146345 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:09:42,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:09:42,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148691 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14614::WARNING::2013-03-17 21:09:49,639::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:09:49,644::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:09:51,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:09:51,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113364 s, 3.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:09:52,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:09:52,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868145 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:09:59,648::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:09:59,649::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:10:00,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:10:00,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:10:00,069::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:10:01,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:10:01,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136586 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:10:02,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:10:02,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973606 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:10:11,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:10:11,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165613 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:10:12,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:10:12,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839522 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:10:21,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:10:21,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158213 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:10:23,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:10:23,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749193 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:10:31,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:10:31,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163831 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:10:33,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:10:33,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170811 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:10:41,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:10:41,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867504 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:10:43,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:10:43,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140536 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:10:51,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:10:51,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105588 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:10:53,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:10:53,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146649 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14615::WARNING::2013-03-17 21:11:00,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:11:00,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:11:01,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:01,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693178 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:11:03,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:11:03,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163954 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:11:10,111::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:11:10,113::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:11:10,513::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:11:10,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:11:10,521::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:11,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:11,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130975 s, 313 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:11:13,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:11:13,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160221 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:21,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:21,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790567 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:11:23,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:11:23,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015138 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:31,792::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:11:31,794::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:11:31,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:31,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000780462 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:31,826::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000780462 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:11:31,826::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:11:31,827::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:11:31,828::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:11:31,829::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:31,830::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:31,830::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:31,831::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:31,831::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:31,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:32,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:32,259::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:11:32,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:11:32,262::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:11:32,263::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:11:32,264::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:11:32,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:32,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:32,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:32,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00717812 s, 571 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:11:33,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:11:33,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180818 s, 227 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:42,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:42,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00236674 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:11:43,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:11:43,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146901 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:11:52,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:11:52,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140821 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:11:53,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:11:53,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761277 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:12:02,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:12:02,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865977 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:12:03,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:12:03,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00338016 s, 1.2 MB/s\n'; <rc> = 0 >Thread-14616::WARNING::2013-03-17 21:12:10,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:12:10,550::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:12:12,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:12:13,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083489 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:12:13,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:12:13,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101066 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:12:20,551::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:12:20,553::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:12:20,554::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:12:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:12:20,960::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:12:20,961::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:12:23,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:12:23,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138937 s, 295 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:12:23,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:12:23,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871767 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:12:33,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:12:33,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0271756 s, 151 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:12:33,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:12:33,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226581 s, 181 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:12:43,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:12:43,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142553 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:12:43,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:12:43,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201105 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:12:53,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:12:53,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145138 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:12:53,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:12:53,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145548 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:13:03,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:13:03,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149846 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:03,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:03,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788569 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:13:13,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:13:13,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730359 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:13,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:13,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013919 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14617::WARNING::2013-03-17 21:13:20,988::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:13:20,992::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:13:23,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:13:23,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880194 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:23,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:23,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111626 s, 3.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:13:30,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:13:30,996::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:13:31,413::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:13:31,420::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:13:31,421::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:13:33,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:13:33,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192877 s, 212 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:33,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:33,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890587 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:13:43,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:13:43,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135954 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:43,947::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:13:43,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:13:43,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:43,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121579 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:43,988::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121579 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:13:43,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:13:43,989::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:13:43,990::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:13:43,991::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:13:43,992::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:13:43,992::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:13:43,993::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:13:43,993::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:13:43,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:44,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:44,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:13:44,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:13:44,443::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:13:44,445::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:13:44,445::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:13:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:44,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:44,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:44,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013634 s, 300 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:13:53,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:13:53,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135149 s, 303 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:13:54,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:13:54,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083166 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:14:04,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:14:04,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096125 s, 426 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:14:04,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:14:05,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014346 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:14:14,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:14:14,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134019 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:14:15,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:14:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136699 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:14:24,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:14:24,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008318 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:14:25,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:14:25,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860102 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14618::WARNING::2013-03-17 21:14:31,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:14:31,458::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:14:34,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:14:34,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871861 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:14:35,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:14:35,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073852 s, 5.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:14:41,463::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:14:41,464::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:14:41,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:14:41,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:14:41,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:14:44,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:14:44,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170565 s, 240 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:14:45,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:14:45,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146425 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:14:54,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:14:54,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953649 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:14:55,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:14:55,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00220382 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:15:04,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:15:04,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982682 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:15:05,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:15:05,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164956 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:15:14,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:15:14,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149505 s, 274 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:15:15,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:15:15,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129781 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:15:24,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:15:24,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107662 s, 380 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:15:25,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:15:25,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125333 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:15:34,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:15:34,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200628 s, 204 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:15:35,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:15:35,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802534 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14619::WARNING::2013-03-17 21:15:41,975::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:15:41,982::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:15:44,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:15:44,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849851 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:15:45,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:15:45,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853993 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:15:51,985::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:15:51,986::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:15:52,423::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:15:52,429::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:15:52,431::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:15:54,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:15:54,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141126 s, 290 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:15:55,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:15:55,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0036658 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:04,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:05,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839407 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:16:05,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:05,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101302 s, 404 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:15,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:15,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108553 s, 377 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:16:15,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:15,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876419 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:25,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:25,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878986 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:16:26,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:26,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140493 s, 292 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:35,213::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:16:35,214::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:16:35,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:35,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144446 s, 142 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:35,266::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144446 s, 142 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:16:35,267::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:16:35,267::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:16:35,269::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:16:35,271::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:16:35,272::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:16:35,272::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:16:35,273::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:16:35,273::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:16:35,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:35,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:35,705::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:16:35,706::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:16:35,709::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:16:35,710::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:16:35,710::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:16:35,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:36,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:36,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130858 s, 313 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:36,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:36,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:36,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160662 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:16:46,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:46,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742326 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:46,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:46,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015061 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14620::WARNING::2013-03-17 21:16:52,466::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:16:52,471::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:16:56,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:16:56,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796998 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:16:56,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:16:56,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837546 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:17:02,479::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:17:02,480::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:17:02,872::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:17:02,879::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:17:02,880::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:17:06,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:17:06,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00347735 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:17:06,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:17:06,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145449 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:17:16,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:17:16,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181464 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:17:16,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:17:16,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075893 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:17:26,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:17:26,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144585 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:17:26,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:17:26,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108396 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:17:36,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:17:36,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834455 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:17:36,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:17:36,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910959 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:17:46,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:17:46,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155203 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:17:46,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:17:46,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080761 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:17:56,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:17:56,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151879 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:17:56,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:17:56,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947001 s, 433 kB/s\n'; <rc> = 0 >Thread-14621::WARNING::2013-03-17 21:18:02,913::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:18:02,916::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:18:06,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:06,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123163 s, 3.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:18:07,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:07,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848458 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:18:12,923::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:18:12,924::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:18:12,924::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:18:13,351::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:18:13,357::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:18:13,359::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:16,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:16,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078445 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:18:17,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:17,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00847701 s, 483 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:26,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:26,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875657 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:18:27,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:27,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148938 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:36,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:36,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015981 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:18:37,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:37,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786603 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:46,738::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:18:46,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:18:46,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:46,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152685 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:46,779::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152685 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:18:46,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:18:46,780::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:18:46,782::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:18:46,783::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:46,783::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:46,784::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:46,784::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:46,784::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:46,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:47,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:47,212::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:18:47,213::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:18:47,219::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:18:47,220::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:18:47,220::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:18:47,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:47,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:47,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147212 s, 278 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:47,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:47,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:47,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123912 s, 331 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:18:57,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:18:57,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113831 s, 360 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:18:57,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:18:57,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158473 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:19:07,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:07,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141409 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:07,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:19:07,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689538 s, 5.9 MB/s\n'; <rc> = 0 >Thread-14622::WARNING::2013-03-17 21:19:13,385::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:19:13,390::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:19:17,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:17,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111774 s, 366 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:17,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:19:17,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151381 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:19:23,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:19:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:19:23,869::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:19:23,876::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:19:23,877::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:19:27,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:27,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171005 s, 240 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:27,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:19:27,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156786 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:19:37,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:37,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152365 s, 269 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:37,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:19:37,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188655 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:47,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:48,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:48,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133505 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:48,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087539 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:19:58,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:19:58,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887768 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:19:58,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:19:58,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141426 s, 290 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:20:08,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:20:08,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858009 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:20:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:20:08,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148423 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:20:18,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:20:18,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831776 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:20:18,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:20:18,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141911 s, 289 kB/s\n'; <rc> = 0 >Thread-14623::WARNING::2013-03-17 21:20:23,905::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:20:23,908::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:20:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:20:28,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833664 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:20:28,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:20:28,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801212 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:20:33,913::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:20:33,914::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:20:34,375::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:20:34,381::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:20:34,383::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:20:38,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:20:38,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085309 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:20:38,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:20:38,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171432 s, 239 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:20:48,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:20:48,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140636 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:20:48,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:20:48,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843291 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:20:58,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:20:58,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080265 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:20:58,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:20:58,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831578 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:21:08,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:21:08,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895129 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:08,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:08,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122748 s, 334 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:21:18,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:21:18,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133721 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:18,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:18,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00799801 s, 512 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:21:28,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:21:28,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141146 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:28,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:28,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253363 s, 1.6 MB/s\n'; <rc> = 0 >Thread-14624::WARNING::2013-03-17 21:21:34,412::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:21:34,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:21:38,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:21:38,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856581 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:38,930::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:21:38,932::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:21:38,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:38,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000808852 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:38,959::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000808852 s, 2.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:21:38,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:21:38,959::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:21:38,961::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:21:38,962::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:21:38,963::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:21:38,963::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:21:38,964::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:21:38,964::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:21:38,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:39,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:39,401::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:21:39,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:21:39,405::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:21:39,406::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:21:39,407::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:21:39,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:39,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:39,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:39,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00600738 s, 682 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:21:44,420::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:21:44,421::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:21:44,877::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:21:44,883::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:21:44,885::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:21:48,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:21:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805128 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:49,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:49,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873853 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:21:58,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:21:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884865 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:21:59,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:21:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839289 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:22:08,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:22:08,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141682 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:22:09,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:22:10,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145279 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:22:18,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:22:18,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017986 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:22:20,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:22:20,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163433 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:22:28,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:22:28,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163056 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:22:30,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:22:30,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151459 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:22:38,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:22:38,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830549 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:22:40,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:22:40,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163067 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14625::WARNING::2013-03-17 21:22:44,909::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:22:44,913::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:22:48,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:22:48,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00455539 s, 899 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:22:50,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:22:50,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197715 s, 207 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:22:54,922::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:22:54,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:22:55,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:22:55,337::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:22:55,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:22:58,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:22:58,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172079 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:23:00,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:23:00,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156718 s, 261 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:08,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:08,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815102 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:23:10,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:23:10,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164374 s, 249 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:18,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:18,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130628 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:23:20,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:23:20,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174314 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:28,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:28,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131735 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:23:30,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:23:30,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011277 s, 363 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:38,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:39,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010086 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:23:40,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:23:40,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147457 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:49,010::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:23:49,011::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:23:49,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:49,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012036 s, 1.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:49,051::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012036 s, 1.7 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:23:49,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:23:49,052::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:23:49,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:23:49,057::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:23:49,058::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:23:49,058::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:23:49,058::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:23:49,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:23:49,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:49,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:49,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:23:49,491::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:23:49,494::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:23:49,495::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:23:49,495::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:23:49,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:49,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:23:49,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:49,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865966 s, 473 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:23:50,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:23:50,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153402 s, 267 kB/s\n'; <rc> = 0 >Thread-14626::WARNING::2013-03-17 21:23:55,365::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:23:55,369::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:23:59,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:23:59,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812947 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:24:00,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:24:01,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167377 s, 2.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:24:05,370::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:24:05,371::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:24:05,372::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:24:05,837::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:24:05,844::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:24:05,845::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:24:09,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:24:09,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749567 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:24:11,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:24:11,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191502 s, 214 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:24:20,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:24:20,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878157 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:24:21,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:24:21,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869705 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:24:30,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:24:30,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164441 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:24:31,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:24:31,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151258 s, 271 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:24:40,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:24:40,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00603675 s, 679 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:24:41,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:24:41,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858155 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:24:50,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:24:50,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140737 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:24:51,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:24:51,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153703 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:25:00,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:25:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174269 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:25:01,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:25:01,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279954 s, 1.5 MB/s\n'; <rc> = 0 >Thread-14627::WARNING::2013-03-17 21:25:05,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:25:05,872::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:25:10,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:25:10,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094131 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:25:11,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:25:11,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081543 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:25:15,875::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:25:15,877::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:25:16,286::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:25:16,293::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:25:16,294::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:25:20,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:25:20,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788028 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:25:21,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:25:21,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120675 s, 339 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:25:30,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:25:30,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849121 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:25:31,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:25:31,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00649646 s, 630 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:25:40,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:25:40,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155329 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:25:41,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:25:41,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104989 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:25:50,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:25:50,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00535192 s, 765 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:25:51,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:25:51,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119044 s, 3.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:26:00,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:26:00,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793038 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:01,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:01,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155989 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:26:10,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:26:10,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149136 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:12,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:12,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143991 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14628::WARNING::2013-03-17 21:26:16,321::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:26:16,325::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:26:20,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:26:20,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964868 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:22,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:22,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087302 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:26:26,327::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:26:26,328::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:26:26,743::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:26:26,749::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:26:26,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:26:30,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:26:30,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743695 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:32,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239242 s, 171 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:26:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:26:40,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886981 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:42,329::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:26:42,331::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:26:42,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:42,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00256991 s, 797 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:42,393::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00256991 s, 797 kB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:26:42,393::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:26:42,394::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:26:42,407::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:26:42,408::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:26:42,409::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:26:42,409::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:26:42,410::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:26:42,410::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:26:42,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:42,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:42,824::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:26:42,825::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:26:42,828::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:26:42,829::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:26:42,829::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:26:42,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:43,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:43,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:43,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0554393 s, 73.9 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:26:50,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:26:50,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136432 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:26:53,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:26:53,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150894 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:27:00,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:27:00,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195914 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:27:03,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:27:03,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104159 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:27:10,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:27:10,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119845 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:27:13,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:27:13,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842878 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:27:20,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:27:20,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142952 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:27:23,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:27:23,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845592 s, 4.8 MB/s\n'; <rc> = 0 >Thread-14629::WARNING::2013-03-17 21:27:26,777::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:27:26,780::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:27:30,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:27:30,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928597 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:27:33,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:27:33,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077217 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:27:36,782::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:27:36,784::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:27:37,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:27:37,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:27:37,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:27:40,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:27:40,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168385 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:27:43,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:27:43,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891098 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:27:50,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:27:50,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142301 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:27:53,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:27:53,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143558 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:00,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:00,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826327 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:28:03,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:28:03,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145868 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:10,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:11,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138871 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:28:13,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:28:13,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174386 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:21,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:21,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145722 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:28:24,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:28:24,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011731 s, 349 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:31,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:31,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145273 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:28:34,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:28:34,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206856 s, 2.0 MB/s\n'; <rc> = 0 >Thread-14630::WARNING::2013-03-17 21:28:37,238::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:28:37,242::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:28:41,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:41,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150042 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:28:44,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:28:44,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139755 s, 293 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:28:47,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:28:47,246::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:28:47,696::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:28:47,703::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:28:47,705::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,150::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:28:51,153::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:28:51,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:51,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000781624 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:51,189::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000781624 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:28:51,189::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:28:51,190::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:28:51,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:28:51,193::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,194::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,194::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,195::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:51,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:51,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:28:51,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:28:51,648::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:28:51,650::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:28:51,650::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:28:51,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:52,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:28:52,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:28:52,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129127 s, 317 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:28:54,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:28:54,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816007 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:29:02,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:29:02,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142075 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:29:04,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:29:04,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156653 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:29:12,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:29:12,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081565 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:29:14,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:29:14,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906611 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:29:22,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:29:22,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008254 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:29:24,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:29:24,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115785 s, 354 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:29:32,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:29:32,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128863 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:29:34,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:29:34,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141551 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:29:42,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:29:42,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143651 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:29:44,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:29:44,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961183 s, 4.3 MB/s\n'; <rc> = 0 >Thread-14631::WARNING::2013-03-17 21:29:47,730::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:29:47,734::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:29:52,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:29:52,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820345 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:29:54,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:29:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784792 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:29:57,735::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:29:57,736::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:29:57,737::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:29:58,222::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:29:58,228::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:29:58,230::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:30:02,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:30:02,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165685 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:30:04,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:30:04,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163781 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:30:12,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:30:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890219 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:30:14,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:30:14,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766887 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:30:22,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:30:22,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166131 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:30:24,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:30:25,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074263 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:30:32,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:30:32,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172371 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:30:35,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:30:35,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177618 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:30:42,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:30:42,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900517 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:30:45,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:30:45,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163628 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:30:52,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:30:52,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082323 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:30:55,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:30:55,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133356 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14632::WARNING::2013-03-17 21:30:58,276::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:30:58,288::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:31:02,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:31:02,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771799 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:05,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:05,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112256 s, 3.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:31:08,291::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:31:08,292::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:31:08,758::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:31:08,765::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:31:08,767::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:31:12,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:31:12,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839941 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:15,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:15,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849324 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:31:22,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:31:22,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139624 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:25,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:25,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135004 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:31:32,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:31:32,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836542 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:35,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:35,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263783 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:31:42,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:31:42,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828261 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:45,620::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:31:45,621::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:31:45,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:45,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145262 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:45,670::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145262 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:31:45,672::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:31:45,673::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:31:45,674::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:31:45,675::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:31:45,676::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:31:45,677::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:31:45,677::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:31:45,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:31:45,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:46,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:46,112::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:31:46,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:31:46,116::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:31:46,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:31:46,117::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:31:46,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:46,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:46,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:46,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0090327 s, 453 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:31:52,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:31:52,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876154 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:31:56,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:31:56,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165563 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:32:02,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:32:02,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100012 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:32:06,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:32:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909136 s, 4.5 MB/s\n'; <rc> = 0 >Thread-14633::WARNING::2013-03-17 21:32:08,793::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:32:08,798::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:32:12,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:32:12,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812299 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:32:16,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:32:16,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794273 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:32:18,846::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:32:18,847::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:32:19,293::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:32:19,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:32:19,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:32:22,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:32:22,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790077 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:32:26,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:32:26,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079713 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:32:32,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:32:32,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151098 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:32:36,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:32:36,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151845 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:32:42,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:32:42,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778694 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:32:46,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:32:47,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148789 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:32:53,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:32:53,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775339 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:32:57,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:32:57,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173465 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:03,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:03,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136279 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:33:07,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:33:07,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014536 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:13,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:13,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108326 s, 3.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:33:17,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:33:17,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700517 s, 5.8 MB/s\n'; <rc> = 0 >Thread-14634::WARNING::2013-03-17 21:33:19,330::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:33:19,336::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:33:23,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:23,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112219 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:33:27,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:33:27,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865567 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:33:29,339::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:33:29,340::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:33:29,766::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:33:29,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:33:29,774::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:33,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:33,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169957 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:33:37,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:33:37,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767453 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:43,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:43,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991882 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:33:47,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:33:47,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154928 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:53,248::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:33:53,250::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:33:53,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:53,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000812999 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:53,281::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000812999 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:33:53,281::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:33:53,282::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:33:53,287::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:33:53,288::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:53,289::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:53,289::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:53,289::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:53,290::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:53,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:53,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:53,716::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:33:53,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:33:53,719::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:33:53,721::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:33:53,721::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:33:53,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:54,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:33:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:33:54,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141641 s, 289 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:33:57,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:33:57,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250779 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:34:04,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:34:04,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739404 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:34:07,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:34:07,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009169 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:34:14,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:34:14,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00820997 s, 499 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:34:17,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:34:17,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016525 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:34:24,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:34:24,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120703 s, 339 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:34:27,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:34:27,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00382226 s, 1.1 MB/s\n'; <rc> = 0 >Thread-14635::WARNING::2013-03-17 21:34:29,797::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:34:29,801::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:34:34,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:34:34,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959294 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:34:37,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:34:37,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814731 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:34:39,804::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:34:39,805::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:34:40,265::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:34:40,272::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:34:40,273::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:34:44,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:34:44,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136637 s, 300 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:34:47,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:34:48,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821913 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:34:54,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:34:54,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789912 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:34:58,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:34:58,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855387 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:35:04,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:35:04,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110971 s, 369 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:35:08,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:35:08,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750583 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:35:14,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:35:14,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861872 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:35:18,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:35:18,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069052 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:35:24,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:35:24,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141985 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:35:28,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:35:28,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141867 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:35:34,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:35:34,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155846 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:35:38,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:35:38,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159683 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14636::WARNING::2013-03-17 21:35:40,305::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:35:40,312::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:35:44,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:35:44,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832611 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:35:48,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:35:48,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837865 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:35:50,314::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:35:50,316::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:35:50,317::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:35:50,712::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:35:50,719::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:35:50,720::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:35:54,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:35:54,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890239 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:35:58,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:35:58,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014932 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:36:04,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:36:04,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861064 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:08,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:08,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00506325 s, 809 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:36:14,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:36:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148517 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:18,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:18,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726437 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:36:24,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:36:24,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169427 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:28,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:28,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151416 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:36:34,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:36:34,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179875 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:38,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792071 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:36:44,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:36:44,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835955 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:48,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:36:48,903::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:36:48,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:48,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000685804 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:48,942::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000685804 s, 3.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:36:48,942::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:36:48,943::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:36:48,944::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:36:48,945::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:36:48,946::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:36:48,947::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:36:48,947::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:36:48,947::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:36:48,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:49,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:49,550::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:36:49,552::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:36:49,555::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:36:49,557::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:36:49,557::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:36:49,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:50,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:36:50,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:36:50,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203927 s, 201 kB/s\n'; <rc> = 0 >Thread-14637::WARNING::2013-03-17 21:36:50,748::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:36:50,752::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:36:54,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:36:54,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889322 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:37:00,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-21::DEBUG::2013-03-17 21:37:00,785::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:37:00,788::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 21:37:00,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017063 s, 240 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:37:01,230::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:37:01,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:37:01,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:37:04,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:37:04,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903548 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:37:10,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:37:10,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816765 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:37:14,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:37:15,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145878 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:37:20,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:37:20,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828076 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:37:25,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:37:25,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170535 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:37:31,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:37:31,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840738 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:37:35,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:37:35,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783799 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:37:41,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:37:41,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00308523 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:37:45,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:37:45,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815763 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:37:51,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:37:51,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136221 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:37:55,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:37:55,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774784 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:38:01,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-14638::WARNING::2013-03-17 21:38:01,288::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:38:01,295::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:38:01,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00793251 s, 516 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:05,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:05,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079812 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:38:11,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:38:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 21:38:11,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:38:11,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00973576 s, 421 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:38:11,750::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:38:11,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:38:11,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:15,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:15,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780014 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:38:21,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:38:21,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219382 s, 187 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:25,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:25,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826539 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:38:31,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:38:31,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108075 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:35,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112947 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:38:41,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:38:41,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891079 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:45,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:45,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786454 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:38:51,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:38:51,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839181 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:55,390::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:38:55,391::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:38:55,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:55,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129684 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:55,434::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129684 s, 1.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:38:55,435::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:38:55,435::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:38:55,437::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:38:55,438::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:55,438::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:55,439::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:55,439::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:55,439::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:55,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:55,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:55,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:38:55,865::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:38:55,868::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:38:55,869::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:38:55,870::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:38:55,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:56,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:38:56,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:38:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188193 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:39:01,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:01,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837547 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:39:06,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:39:06,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133745 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14639::WARNING::2013-03-17 21:39:11,785::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:39:11,789::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:39:11,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:11,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870947 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:39:16,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:39:16,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857812 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:39:21,791::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:39:21,792::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:21,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:22,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147581 s, 2.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:39:22,293::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:39:22,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:39:22,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:39:26,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:39:26,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717059 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:39:32,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:32,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747748 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:39:36,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:39:36,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856034 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:39:42,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:42,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00326261 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:39:46,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:39:46,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139778 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:39:52,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:39:52,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832816 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:39:56,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:39:56,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825748 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:40:02,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:02,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164368 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:40:06,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:40:06,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827038 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:40:12,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:12,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143343 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:40:16,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:40:16,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144763 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14640::WARNING::2013-03-17 21:40:22,336::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:40:22,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:40:22,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:22,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859328 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:40:26,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:40:26,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839035 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:40:32,348::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:40:32,349::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:32,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:32,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168983 s, 242 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:40:32,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:40:32,835::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:40:32,837::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:40:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:40:36,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883162 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:40:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:42,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837385 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:40:46,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:40:46,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161167 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:40:52,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:40:52,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161262 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:40:56,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:40:56,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081897 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:02,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:02,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265448 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:41:06,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:41:06,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139177 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:12,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:12,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844244 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:41:16,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:41:16,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167426 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:22,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:22,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136735 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:41:26,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:41:26,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171159 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14641::WARNING::2013-03-17 21:41:32,866::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:41:32,869::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:41:33,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:33,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812215 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:41:36,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:41:36,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781515 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:41:42,871::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:41:42,872::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:41:42,873::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:43,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:43,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144183 s, 284 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:41:43,411::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:41:43,417::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:41:43,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:41:46,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:41:46,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746858 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:53,197::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:41:53,199::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:41:53,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:53,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0011025 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:53,232::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0011025 s, 1.9 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:41:53,233::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:41:53,233::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:41:53,234::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:41:53,235::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:41:53,236::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:41:53,237::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:41:53,237::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:41:53,237::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:41:53,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:53,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:53,699::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:41:53,700::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:41:53,703::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:41:53,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:41:53,706::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:41:53,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:54,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:41:54,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:41:54,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159957 s, 256 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:41:56,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:41:56,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905704 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:42:04,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:42:04,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126012 s, 325 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:42:06,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:42:07,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160067 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:42:14,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:42:14,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835096 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:42:17,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:42:17,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927854 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:42:24,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:42:24,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095008 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:42:27,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:42:27,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140011 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:42:34,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:42:34,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153008 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:42:37,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:42:37,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881085 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14642::WARNING::2013-03-17 21:42:43,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:42:43,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:42:44,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:42:44,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815318 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:42:47,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:42:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788332 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:42:53,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:42:53,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:42:53,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:42:53,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:42:53,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:42:54,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:42:54,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917717 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:42:57,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:42:57,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795961 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:43:04,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:43:04,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000696458 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:07,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:07,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135362 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:43:14,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:43:14,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084464 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:17,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:17,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799716 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:43:24,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:43:24,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826769 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:27,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:27,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139135 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:43:34,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:43:34,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139926 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:37,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:37,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148237 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:43:44,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:43:45,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149237 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:47,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:47,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151705 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14643::WARNING::2013-03-17 21:43:53,976::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:43:53,979::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:43:55,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:43:55,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837448 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:57,453::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:43:57,456::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:43:57,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:57,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000804325 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:57,485::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000804325 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:43:57,485::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:43:57,485::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:43:57,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:43:57,488::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:43:57,489::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:43:57,489::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:43:57,490::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:43:57,490::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:43:57,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:57,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:57,908::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:43:57,909::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:43:57,911::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:43:57,912::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:43:57,913::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:43:57,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:58,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:43:58,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:43:58,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00470661 s, 870 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:44:03,981::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:44:03,983::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:44:04,383::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:44:04,393::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:44:04,394::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:44:05,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:44:05,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091602 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:44:08,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:44:08,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909446 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:44:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:44:15,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884929 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:44:18,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:44:18,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846618 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:44:25,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:44:25,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886174 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:44:28,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:44:28,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826612 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:44:35,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:44:35,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858286 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:44:38,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:44:38,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157844 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:44:45,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:44:45,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824465 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:44:48,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:44:48,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143293 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:44:55,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:44:55,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074528 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:44:58,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:44:58,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170762 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14644::WARNING::2013-03-17 21:45:04,438::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:45:04,445::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:45:05,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:45:05,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728556 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:45:08,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:45:08,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994414 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:45:14,447::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:45:14,449::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:45:14,911::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:45:14,918::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:45:14,919::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:45:15,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:45:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169744 s, 241 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:45:18,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:45:18,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814222 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:45:25,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:45:25,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893446 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:45:28,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:45:28,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911021 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:45:35,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:45:35,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778109 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:45:38,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:45:38,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142498 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:45:45,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:45:45,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165306 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:45:48,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:45:48,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137116 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:45:55,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:45:55,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168765 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:45:58,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:45:58,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140622 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:05,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:06,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143498 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:46:08,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:46:08,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0024399 s, 1.7 MB/s\n'; <rc> = 0 >Thread-14645::WARNING::2013-03-17 21:46:14,946::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:46:14,950::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:46:16,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:16,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912412 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:46:18,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:46:18,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898426 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:46:24,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:46:24,954::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:46:25,387::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:46:25,393::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:46:25,395::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:26,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:26,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171745 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:46:28,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:46:28,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751535 s, 5.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:36,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:36,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715456 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:46:38,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:46:38,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144497 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:46,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:46,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847606 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:46:48,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:46:48,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818756 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:56,356::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:46:56,358::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:46:56,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:56,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133212 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:56,401::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133212 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:46:56,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:46:56,402::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:46:56,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:46:56,404::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:56,405::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:56,406::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:56,406::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:56,406::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:56,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:56,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:56,806::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:46:56,808::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:46:56,812::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:46:56,813::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:46:56,814::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:46:56,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:57,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:46:57,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:46:57,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112596 s, 364 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:46:59,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:46:59,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145543 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:47:07,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:47:07,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168711 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:47:09,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:47:09,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165257 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:47:17,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:47:17,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969586 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:47:19,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:47:19,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162798 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14646::WARNING::2013-03-17 21:47:25,420::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:47:25,424::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:47:27,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:47:27,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812278 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:47:29,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:47:29,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844112 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:47:35,426::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:47:35,427::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:47:35,428::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:47:35,870::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:47:35,876::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:47:35,877::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:47:37,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:47:37,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101285 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:47:39,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:47:39,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143549 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:47:47,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:47:47,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830875 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:47:49,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:47:49,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133899 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:47:57,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:47:57,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862349 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:47:59,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:47:59,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192044 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:48:07,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:48:07,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138903 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:09,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:09,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926172 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:48:17,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:48:17,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793323 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:19,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:19,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166015 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:48:27,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:48:28,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094657 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:29,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:29,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075314 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14647::WARNING::2013-03-17 21:48:35,906::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:48:35,909::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:48:38,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:48:38,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895345 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:39,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:39,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852047 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:48:45,911::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:48:45,912::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:48:46,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:48:46,338::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:48:46,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:48:48,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:48:48,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810164 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:49,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:49,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730187 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:48:58,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:48:58,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092239 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:59,520::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:48:59,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:48:59,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:59,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000984777 s, 2.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:59,556::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000984777 s, 2.1 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:48:59,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:48:59,557::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:48:59,559::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:48:59,560::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:48:59,560::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:48:59,561::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:48:59,561::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:48:59,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:48:59,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:48:59,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:48:59,998::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:48:59,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:49:00,001::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:49:00,002::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:49:00,003::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:49:00,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:00,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:49:00,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:00,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186433 s, 220 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:49:08,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:49:08,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153042 s, 268 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:49:10,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:10,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148308 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:49:18,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:49:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892006 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:49:20,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:20,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150615 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:49:28,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:49:28,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896464 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:49:30,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:30,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775272 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:49:38,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:49:38,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155801 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:49:40,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:40,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078927 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14648::WARNING::2013-03-17 21:49:46,365::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:49:46,373::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:49:48,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:49:48,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886822 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:49:50,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:49:50,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076054 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:49:56,376::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:49:56,377::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:49:56,922::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:49:56,929::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:49:56,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:49:58,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:49:58,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080472 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:50:00,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:50:00,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908911 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:50:08,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:50:08,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891626 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:50:10,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:50:10,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147577 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:50:18,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:50:18,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828572 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:50:20,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:50:20,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094585 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:50:28,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:50:28,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145431 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:50:30,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:50:30,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142393 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:50:38,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:50:38,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151013 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:50:40,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:50:40,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164738 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:50:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:50:49,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144244 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:50:50,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:50:50,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083195 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14649::WARNING::2013-03-17 21:50:56,955::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:50:56,958::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:50:59,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:50:59,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853948 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:51:00,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:51:00,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106878 s, 3.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:51:06,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:51:06,962::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:51:07,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:51:07,378::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:51:07,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:51:09,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:51:09,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873292 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:51:10,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:51:10,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147678 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:51:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:51:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146392 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:51:20,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:51:21,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905103 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:51:29,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:51:29,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001643 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:51:31,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:51:31,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165344 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:51:39,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:51:39,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868867 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:51:41,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:51:41,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133313 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:51:49,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:51:49,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154139 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:51:51,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:51:51,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135264 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:51:59,567::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:51:59,569::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:51:59,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:51:59,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00165408 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:51:59,602::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00165408 s, 1.2 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:51:59,603::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:51:59,603::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:51:59,604::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:51:59,608::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:51:59,609::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:51:59,609::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:51:59,610::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:51:59,610::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:51:59,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:00,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:52:00,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:52:00,142::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:52:00,144::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:52:00,145::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:52:00,146::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:52:00,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:00,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:52:00,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:00,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215093 s, 190 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:52:01,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:52:01,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132613 s, 309 kB/s\n'; <rc> = 0 >Thread-14650::WARNING::2013-03-17 21:52:07,405::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:52:07,409::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:52:10,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:10,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747595 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:52:11,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:52:11,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790792 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:52:17,411::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:52:17,413::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:52:17,814::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:52:17,821::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:52:17,823::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:52:20,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:20,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106738 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:52:21,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:52:21,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839774 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:52:30,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:30,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142715 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:52:31,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:52:31,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154159 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:52:40,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:40,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821744 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:52:41,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:52:41,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144359 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:52:50,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:52:50,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123766 s, 331 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:52:51,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:52:51,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145152 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:53:01,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:53:01,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992175 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:53:01,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:53:01,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145198 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:53:11,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:53:11,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175547 s, 233 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:53:11,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:53:11,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777514 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14651::WARNING::2013-03-17 21:53:17,856::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:53:17,860::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:53:21,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:53:21,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891733 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:53:21,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:53:21,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196538 s, 2.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:53:27,861::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:53:27,862::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:53:27,863::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:53:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:53:28,256::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:53:28,257::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:53:31,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:53:31,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907834 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:53:31,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:53:31,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146441 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:53:41,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:53:41,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909289 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:53:41,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:53:41,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189331 s, 2.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:53:51,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:53:51,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816741 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:53:51,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:53:51,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135887 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:54:01,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:54:01,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075659 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:01,670::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:54:01,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:54:01,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:01,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00532055 s, 385 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:01,733::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00532055 s, 385 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:54:01,734::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:54:01,734::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:54:01,736::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:54:01,737::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:54:01,738::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:54:01,738::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:54:01,739::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:54:01,739::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:54:01,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:02,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:02,271::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:54:02,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:54:02,275::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:54:02,276::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:54:02,276::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:54:02,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:02,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:02,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:02,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015012 s, 273 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:54:11,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:54:11,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139019 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:12,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:12,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898064 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:54:21,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:54:21,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716031 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:22,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:22,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139391 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14652::WARNING::2013-03-17 21:54:28,290::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:54:28,295::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:54:31,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:54:31,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185025 s, 2.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:32,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:32,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828122 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:54:38,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:54:38,302::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:54:38,701::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:54:38,708::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:54:38,710::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:54:41,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:54:41,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873635 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:42,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:42,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766474 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:54:51,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:54:51,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150884 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:54:52,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:54:52,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161642 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:55:01,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:55:01,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842287 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:55:02,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:55:02,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802995 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:55:12,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:55:12,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180963 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:55:12,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:55:12,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133782 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:55:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:55:22,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148873 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:55:22,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:55:22,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160283 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:55:32,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:55:32,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923323 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:55:32,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:55:33,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136752 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14653::WARNING::2013-03-17 21:55:38,736::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:55:38,739::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:55:42,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:55:42,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923639 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:55:43,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:55:43,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846994 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:55:48,741::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:55:48,743::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:55:49,142::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:55:49,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:55:49,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:55:52,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:55:52,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841318 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:55:53,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:55:53,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862295 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:56:02,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:56:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815936 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:56:03,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:56:03,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241652 s, 170 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:56:12,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:56:12,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862276 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:56:13,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:56:13,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080674 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:56:22,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:56:22,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146962 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:56:23,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:56:23,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080547 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:56:32,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:56:32,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782281 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:56:33,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:56:33,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00248682 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:56:42,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:56:42,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983052 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:56:43,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:56:43,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013934 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14654::WARNING::2013-03-17 21:56:49,185::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:56:49,190::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:56:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:56:52,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872664 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:56:53,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:56:53,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128217 s, 3.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:56:59,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:56:59,196::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:56:59,633::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:56:59,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:56:59,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:02,856::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 21:57:02,859::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:57:02,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:02,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140403 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:02,897::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140403 s, 1.5 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 21:57:02,898::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 21:57:02,898::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 21:57:02,900::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 21:57:02,901::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:02,901::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:02,902::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:02,902::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:02,902::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:02,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:03,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:03,327::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:57:03,328::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 21:57:03,330::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 21:57:03,331::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 21:57:03,332::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 21:57:03,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:03,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:03,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00744637 s, 550 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:03,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:03,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:03,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00765643 s, 535 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:57:13,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:13,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115878 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:13,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:13,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780742 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:57:23,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:23,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863426 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:23,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:23,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801453 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:57:33,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:33,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177542 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:34,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:34,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144981 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:57:43,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982513 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:44,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:44,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118354 s, 346 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:57:53,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:57:53,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115804 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:57:54,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:57:54,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114423 s, 358 kB/s\n'; <rc> = 0 >Thread-14655::WARNING::2013-03-17 21:57:59,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:57:59,677::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 21:58:03,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:58:03,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125666 s, 326 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:58:04,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:58:04,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868141 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:58:09,680::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:58:09,681::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:58:10,149::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:58:10,156::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:58:10,157::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:58:13,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:58:13,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776539 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:58:14,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:58:14,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134097 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:58:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:58:23,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174437 s, 2.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:58:24,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:58:24,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.009767 s, 419 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:58:33,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:58:33,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141781 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:58:34,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:58:34,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167677 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:58:43,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:58:44,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107 s, 383 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:58:44,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:58:44,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153422 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:58:54,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:58:54,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016493 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:58:54,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:58:54,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844434 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:04,085::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 21:59:04,088::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:59:04,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:04,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00923854 s, 222 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:04,128::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00923854 s, 222 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 21:59:04,129::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 21:59:04,129::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 21:59:04,132::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 21:59:04,133::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:59:04,133::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:59:04,134::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:59:04,134::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:59:04,135::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 21:59:04,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:04,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:04,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 21:59:04,590::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 21:59:04,592::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 21:59:04,593::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 21:59:04,594::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 21:59:04,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:04,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:04,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136378 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:05,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:05,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:05,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185998 s, 220 kB/s\n'; <rc> = 0 >Thread-14656::WARNING::2013-03-17 21:59:10,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 21:59:10,193::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 21:59:14,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:14,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913014 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:15,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:15,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962394 s, 4.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 21:59:20,195::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 21:59:20,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 21:59:20,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 21:59:20,629::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 21:59:20,636::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 21:59:20,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 21:59:24,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:24,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816533 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:25,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:25,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765397 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:59:34,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:35,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015269 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:35,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:35,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887134 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:59:45,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:45,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139483 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:45,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:45,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079002 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 21:59:55,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 21:59:55,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736714 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 21:59:55,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 21:59:55,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859778 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:00:05,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:00:05,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:00:05,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010297 s, 398 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:00:05,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101653 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:00:15,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:00:15,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:00:15,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112567 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:00:15,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141968 s, 2.9 MB/s\n'; <rc> = 0 >Thread-14657::WARNING::2013-03-17 22:00:20,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:00:20,671::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:00:25,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:00:25,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791871 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:00:25,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:00:25,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815811 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:00:30,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:00:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:00:31,098::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:00:31,105::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:00:31,106::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:00:35,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:00:35,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014893 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:00:35,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:00:35,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791313 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:00:45,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:00:45,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077517 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:00:45,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:00:45,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017117 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:00:55,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:00:55,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227701 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:00:55,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:00:55,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812497 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:01:05,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:01:05,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158028 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:01:05,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:01:05,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157637 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:01:15,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:01:15,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177146 s, 231 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:01:15,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:01:15,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115805 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:01:25,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:01:25,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137621 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:01:25,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:01:26,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875117 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14658::WARNING::2013-03-17 22:01:31,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:01:31,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:01:35,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:01:35,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095384 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:01:36,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:01:36,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201378 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:01:41,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:01:41,141::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:01:41,580::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:01:41,587::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:01:41,588::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:01:45,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:01:45,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151797 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:01:46,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:01:46,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823519 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:01:55,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:01:55,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827973 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:01:56,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:01:56,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165201 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:02:05,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:02:05,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00271824 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:06,305::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:02:06,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:02:06,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:06,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0008569 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:06,341::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0008569 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:02:06,341::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:02:06,342::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:02:06,344::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:02:06,345::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:02:06,345::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:02:06,346::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:02:06,346::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:02:06,347::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:02:06,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:06,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:06,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:02:06,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:02:06,806::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:02:06,807::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:02:06,807::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:02:06,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:07,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:07,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:07,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132503 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:02:15,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:02:16,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923433 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:17,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:17,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162762 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:02:26,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:02:26,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157949 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:27,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:27,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972102 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:02:36,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:02:36,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00330456 s, 1.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:37,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:37,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086558 s, 4.7 MB/s\n'; <rc> = 0 >Thread-14659::WARNING::2013-03-17 22:02:41,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:02:41,618::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:02:46,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:02:46,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082339 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:47,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:47,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164246 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:02:51,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:02:51,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:02:52,056::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:02:52,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:02:52,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:02:56,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:02:56,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838952 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:02:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:02:57,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870287 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:03:06,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:03:06,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164504 s, 249 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:03:07,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:03:07,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171789 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:03:16,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:03:16,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151316 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:03:17,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:03:17,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127699 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:03:26,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:03:26,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911906 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:03:27,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:03:27,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998689 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:03:36,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:03:36,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154354 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:03:37,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:03:37,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170266 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:03:46,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:03:46,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101838 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:03:47,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:03:47,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765593 s, 5.4 MB/s\n'; <rc> = 0 >Thread-14660::WARNING::2013-03-17 22:03:52,089::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:03:52,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:03:56,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:03:56,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892024 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:03:57,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:03:58,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989489 s, 4.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:04:02,094::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:04:02,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:04:02,556::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:04:02,562::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:04:02,564::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,454::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:04:06,457::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:04:06,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:06,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00153085 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:06,497::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00153085 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:04:06,498::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:04:06,498::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:04:06,499::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:04:06,501::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,501::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,502::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,502::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,502::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:06,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:06,914::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:04:06,915::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:04:06,918::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:04:06,919::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:04:06,919::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:04:06,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:07,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:07,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:07,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0317966 s, 129 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:04:08,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:04:08,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0069208 s, 592 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:17,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:17,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136889 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:04:18,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:04:18,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832502 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:27,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862565 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:04:28,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:04:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707758 s, 5.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:37,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:37,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112329 s, 3.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:04:38,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:04:38,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153865 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:47,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:47,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803359 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:04:48,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:04:48,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083374 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:04:57,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:04:57,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166873 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:04:58,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:04:58,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840369 s, 4.9 MB/s\n'; <rc> = 0 >Thread-14661::WARNING::2013-03-17 22:05:02,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:05:02,597::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:05:07,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:05:07,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806055 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:05:08,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:05:08,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847484 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:05:12,600::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:05:12,601::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:05:12,601::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:05:13,037::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:05:13,044::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:05:13,046::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:05:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:05:17,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192802 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:05:18,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:05:18,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761652 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:05:27,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:05:27,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852082 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:05:28,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:05:28,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740987 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:05:37,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:05:37,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147622 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:05:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:05:38,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853311 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:05:47,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:05:47,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777364 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:05:48,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:05:48,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105916 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:05:57,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:05:57,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963385 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:05:58,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:05:58,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00417839 s, 980 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:06:07,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:06:07,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754915 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:06:08,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:06:08,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013375 s, 3.1 MB/s\n'; <rc> = 0 >Thread-14662::WARNING::2013-03-17 22:06:13,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:06:13,074::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:06:18,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:06:18,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831022 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:06:19,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:06:19,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733752 s, 5.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:06:23,077::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:06:23,078::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:06:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:06:23,579::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:06:23,581::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:06:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:06:28,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00283033 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:06:29,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:06:29,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960864 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:06:38,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:06:38,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729917 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:06:39,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:06:39,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847413 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:06:48,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:06:48,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862158 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:06:49,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:06:49,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152257 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:06:58,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:06:58,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160994 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:06:59,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:06:59,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174543 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:07:08,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:07:08,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713886 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:09,412::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:07:09,414::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:07:09,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:09,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00147914 s, 1.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:09,458::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00147914 s, 1.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:07:09,459::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:07:09,459::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:07:09,461::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:07:09,463::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:07:09,464::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:07:09,464::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:07:09,464::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:07:09,465::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:07:09,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:09,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:09,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:07:09,901::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:07:09,903::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:07:09,905::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:07:09,905::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:07:09,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:10,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:10,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016594 s, 247 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:07:18,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:07:18,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845765 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:20,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:20,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767711 s, 5.3 MB/s\n'; <rc> = 0 >Thread-14663::WARNING::2013-03-17 22:07:23,605::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:07:23,609::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:07:28,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:07:28,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117335 s, 349 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:30,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832507 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:07:33,612::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:07:33,613::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:07:34,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:07:34,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:07:34,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:07:38,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:07:38,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152087 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:40,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:40,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154171 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:07:48,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:07:48,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134486 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:07:50,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:07:50,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175515 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:07:58,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:07:58,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141599 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:08:00,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:08:00,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084587 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:08:08,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:08:08,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761464 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:08:10,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:08:10,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820646 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:08:18,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:08:18,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916927 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:08:20,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:08:20,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145087 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:08:28,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:08:28,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154822 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:08:30,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:08:30,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144622 s, 2.8 MB/s\n'; <rc> = 0 >Thread-14664::WARNING::2013-03-17 22:08:34,104::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:08:34,108::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:08:38,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:08:38,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773098 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:08:41,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:08:41,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872241 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:08:44,110::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:08:44,111::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:08:44,521::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:08:44,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:08:44,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:08:48,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:08:48,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916112 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:08:51,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:08:51,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854181 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:08:58,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:08:58,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154176 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:09:01,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:09:01,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819937 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:08,962::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:09:08,963::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:09:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:09,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00155602 s, 1.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:09,003::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00155602 s, 1.3 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:09:09,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:09:09,004::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:09:09,006::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:09:09,007::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:09:09,007::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:09:09,008::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:09:09,008::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:09:09,009::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:09:09,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:09,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:09,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:09:09,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:09:09,444::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:09:09,445::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:09:09,445::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:09:09,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:09,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:09,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:09,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155111 s, 264 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:09:11,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:09:11,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017679 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:19,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:19,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875435 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:09:21,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:09:21,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147618 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:29,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:29,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152136 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:09:31,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:09:31,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155759 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:09:39,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:40,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844068 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:09:41,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:09:41,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173188 s, 2.4 MB/s\n'; <rc> = 0 >Thread-14665::WARNING::2013-03-17 22:09:44,559::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:09:44,562::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:09:50,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:09:50,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157582 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:09:51,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:09:51,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121907 s, 3.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:09:54,563::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:09:54,564::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:09:55,019::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:09:55,026::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:09:55,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:10:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:10:00,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143386 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:10:01,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:10:01,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00401416 s, 1.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:10:10,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:10:10,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00318727 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:10:11,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:10:11,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940289 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:10:20,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:10:20,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863652 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:10:21,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:10:21,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154157 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:10:30,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:10:30,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811971 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:10:31,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:10:31,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851316 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:10:40,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:10:40,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156609 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:10:42,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:10:42,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956744 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:10:50,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:10:50,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157126 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:10:52,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:10:52,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160495 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14666::WARNING::2013-03-17 22:10:55,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:10:55,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:11:00,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:11:00,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835333 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:11:02,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:11:02,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927042 s, 4.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:11:05,069::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:11:05,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:11:05,071::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:11:05,503::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:11:05,510::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:11:05,511::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:11:10,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:11:10,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145082 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:11:12,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:11:12,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148927 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:11:20,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:11:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705265 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:11:22,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:11:22,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147146 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:11:30,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:11:30,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770329 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:11:32,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:11:32,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745543 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:11:40,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:11:40,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856516 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:11:42,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:11:42,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826016 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:11:50,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:11:50,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774622 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:11:52,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:11:52,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749116 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:12:00,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:12:00,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144086 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:02,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:02,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163067 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14667::WARNING::2013-03-17 22:12:05,535::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:12:05,539::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:12:10,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:12:10,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796957 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:12,748::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:12:12,749::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:12:12,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:12,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000767506 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:12,784::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000767506 s, 2.7 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:12:12,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:12:12,784::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:12:12,786::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:12:12,788::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:12:12,788::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:12:12,789::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:12:12,789::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:12:12,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:12:12,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:13,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:13,206::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:12:13,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:12:13,210::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:12:13,211::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:12:13,211::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:12:13,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:13,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:13,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:13,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00730073 s, 561 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:12:15,541::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:12:15,542::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:12:15,964::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:12:15,971::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:12:15,972::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:12:20,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:12:20,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101969 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:23,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:23,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883127 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:12:30,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:12:30,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796008 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:33,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:33,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886171 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:12:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:12:40,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015188 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:43,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:43,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857384 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:12:50,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:12:50,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786354 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:12:53,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:12:53,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740469 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:13:00,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:13:00,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157595 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:13:03,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:13:03,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917309 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:13:10,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:13:10,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764974 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:13:14,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:13:14,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162373 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14668::WARNING::2013-03-17 22:13:16,001::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:13:16,005::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:13:20,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:13:20,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824699 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:13:24,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:13:24,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100141 s, 409 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:13:26,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:13:26,009::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:13:26,447::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:13:26,454::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:13:26,455::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:13:31,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:13:31,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121202 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:13:34,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:13:34,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854006 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:13:41,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:13:41,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135349 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:13:44,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:13:44,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0029965 s, 1.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:13:51,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:13:51,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826633 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:13:54,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:13:54,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069261 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:01,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:01,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132561 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:14:04,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:14:04,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168875 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:11,158::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:14:11,160::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:14:11,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:11,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00069519 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:11,194::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00069519 s, 2.9 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:14:11,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:14:11,195::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:14:11,197::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:14:11,197::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:11,198::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:11,198::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:11,199::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:11,199::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:11,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:11,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:11,607::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:11,608::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:14:11,613::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:14:11,614::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:14:11,614::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:14:11,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:11,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:11,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:12,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182572 s, 224 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:14:14,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:14:14,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921498 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:22,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:22,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157658 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:14:24,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:14:24,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740256 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14669::WARNING::2013-03-17 22:14:26,494::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:14:26,497::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:14:32,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:32,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898188 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:14:34,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:14:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879221 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:14:36,499::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:14:36,500::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:14:36,955::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:14:36,962::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:14:36,963::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:14:42,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:42,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269121 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:14:44,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:14:44,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141706 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:14:52,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:14:52,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798776 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:14:54,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:14:54,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175886 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:15:02,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:15:02,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778291 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:15:04,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:15:04,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841453 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:15:12,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:15:12,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143628 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:15:14,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:15:14,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789131 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:15:22,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:15:22,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154624 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:15:25,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:15:25,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0344521 s, 119 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:15:32,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:15:32,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129296 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:15:35,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:15:35,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111792 s, 3.7 MB/s\n'; <rc> = 0 >Thread-14670::WARNING::2013-03-17 22:15:36,994::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:15:36,997::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:15:42,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:15:42,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935877 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:15:45,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:15:45,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871103 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:15:46,999::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:15:47,000::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:15:47,448::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:15:47,455::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:15:47,457::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:15:52,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:15:52,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955067 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:15:55,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:15:55,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849065 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:16:02,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:16:02,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154211 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:16:05,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:16:05,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147342 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:16:12,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:16:12,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964352 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:16:15,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:16:15,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137323 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:16:22,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:16:22,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858452 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:16:25,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:16:25,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143477 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:16:32,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:16:32,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140555 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:16:35,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:16:35,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144517 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:16:42,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:16:42,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808458 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:16:45,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:16:45,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015919 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14671::WARNING::2013-03-17 22:16:47,492::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:16:47,496::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:16:52,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:16:52,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804547 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:16:55,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:16:55,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836492 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:16:57,497::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:16:57,500::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:16:57,501::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:16:57,927::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:16:57,934::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:16:57,935::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:17:02,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:17:02,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845497 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:05,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:05,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013845 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:17:12,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:17:12,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081592 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:15,948::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:17:15,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:17:15,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:15,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000783255 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:15,985::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000783255 s, 2.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:17:15,986::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:17:15,986::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:17:15,988::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:17:15,990::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:17:15,991::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:17:15,991::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:17:15,992::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:17:15,992::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:17:15,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:16,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:16,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:17:16,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:17:16,384::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:17:16,385::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:17:16,385::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:17:16,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:16,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:16,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:16,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159493 s, 257 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:17:22,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:17:22,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952184 s, 4.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:26,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:26,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753577 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:17:32,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:17:32,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804412 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:36,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:36,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143311 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:17:42,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:17:42,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700548 s, 5.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:46,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:47,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697567 s, 5.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:17:52,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:17:52,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157231 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:17:57,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:17:57,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.033501 s, 122 kB/s\n'; <rc> = 0 >Thread-14672::WARNING::2013-03-17 22:17:57,965::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:17:57,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:18:02,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:18:02,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839964 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:18:07,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:18:07,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761877 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:18:08,002::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:18:08,003::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:18:08,416::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:18:08,423::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:18:08,424::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:18:13,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:18:13,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715823 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:18:17,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:18:17,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792599 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:18:23,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:18:23,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159145 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:18:27,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:18:27,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154648 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:18:33,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:18:33,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149322 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:18:37,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:18:37,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010035 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:18:43,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:18:43,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146672 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:18:47,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:18:47,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785329 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:18:53,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:18:53,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133322 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:18:57,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:18:57,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156437 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:03,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:03,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0526939 s, 77.7 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:19:07,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:19:08,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940237 s, 4.4 MB/s\n'; <rc> = 0 >Thread-14673::WARNING::2013-03-17 22:19:08,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:19:08,458::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:19:13,314::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:19:13,315::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:19:13,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:13,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000730821 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:13,343::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000730821 s, 2.8 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:19:13,343::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:19:13,343::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:19:13,345::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:19:13,346::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:13,347::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:13,347::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:13,348::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:13,348::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:13,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:13,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:13,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:13,795::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:19:13,798::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:19:13,799::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:19:13,799::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:19:13,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:14,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:14,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:14,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209992 s, 195 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:19:18,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:19:18,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215704 s, 190 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:19:18,466::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:19:18,467::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:19:18,888::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:19:18,896::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:19:18,898::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:19:24,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:24,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00979003 s, 418 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:19:28,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:19:28,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810968 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:34,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115942 s, 3.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:19:38,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:19:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785655 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:44,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:44,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109334 s, 375 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:19:48,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:19:48,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137263 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:19:54,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:19:54,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204222 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:19:58,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:19:58,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142559 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:20:04,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:20:04,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016628 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:20:08,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:08,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942047 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:20:14,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:20:14,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141049 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:20:18,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:18,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750403 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14674::WARNING::2013-03-17 22:20:18,930::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:20:18,941::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:20:24,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:20:24,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757877 s, 5.4 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:20:28,956::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:20:28,957::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:28,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:29,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00933382 s, 439 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:20:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:20:29,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:20:29,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:20:34,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:20:34,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820701 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:20:39,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:39,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900933 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:20:44,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:20:44,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100968 s, 406 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:20:49,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:49,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883497 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:20:54,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:20:54,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081721 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:20:59,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:20:59,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108668 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:21:04,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:21:04,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146075 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:21:09,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:09,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154026 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:21:14,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:21:14,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687547 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:21:19,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166949 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:21:24,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:21:24,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025488 s, 161 kB/s\n'; <rc> = 0 >Thread-14675::WARNING::2013-03-17 22:21:29,451::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:21:29,456::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:21:29,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:29,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823079 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:21:34,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:21:34,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227776 s, 180 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:21:39,458::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:21:39,459::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:39,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:39,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175869 s, 2.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:21:39,925::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:21:39,930::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:21:39,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:21:44,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:21:44,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812481 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:21:49,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:49,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141329 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:21:54,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:21:54,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844851 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:21:59,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:21:59,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724404 s, 5.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:22:04,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:22:04,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819769 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:22:09,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:09,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086635 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:22:14,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:22:15,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379912 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:22:19,843::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:22:19,845::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:22:19,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:19,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000985291 s, 2.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:22:19,892::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000985291 s, 2.1 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:22:19,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:22:19,894::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:22:19,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:22:19,897::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:19,897::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:19,898::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:19,898::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:19,899::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:19,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:20,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:22:20,347::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:20,348::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:22:20,350::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:22:20,351::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:22:20,352::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:22:20,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:20,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:22:20,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:20,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865821 s, 473 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:22:25,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:22:25,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141665 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:22:30,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:30,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014053 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:22:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:22:35,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088236 s, 4.6 MB/s\n'; <rc> = 0 >Thread-14676::WARNING::2013-03-17 22:22:39,958::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:22:39,962::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:22:40,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:40,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156909 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:22:45,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:22:45,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825685 s, 5.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:22:49,963::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:22:49,965::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:22:49,966::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:22:50,488::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:22:50,495::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:22:50,496::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:22:50,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:22:51,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00873286 s, 469 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:22:55,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:22:55,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854382 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:23:01,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:23:01,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151288 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:23:05,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:23:05,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084794 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:23:11,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:23:11,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170016 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:23:15,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:23:15,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079278 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:23:21,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:23:21,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860293 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:23:25,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:23:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149421 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:23:31,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:23:31,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754052 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:23:35,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:23:35,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159991 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:23:41,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:23:41,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925216 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:23:45,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:23:45,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163076 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14677::WARNING::2013-03-17 22:23:50,519::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:23:50,523::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:23:51,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:23:51,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157698 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:23:55,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:23:55,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840471 s, 4.9 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:24:00,525::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:24:00,527::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:24:00,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:24:00,945::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:24:00,946::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:24:01,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:24:01,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146903 s, 279 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:05,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:05,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803617 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:24:11,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:24:11,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757287 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:15,450::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:24:15,451::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:24:15,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:15,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000793312 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:15,479::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000793312 s, 2.6 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:24:15,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:24:15,479::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:24:15,483::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:24:15,484::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:24:15,485::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:24:15,485::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:24:15,486::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:24:15,486::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:24:15,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:15,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:15,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:24:15,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:24:15,924::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:24:15,925::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:24:15,926::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:24:15,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:16,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:16,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:16,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144858 s, 283 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:24:21,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:24:21,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171729 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:26,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:26,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169096 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:24:31,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:24:31,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838735 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:36,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148747 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:24:41,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:24:41,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131005 s, 3.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:46,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:46,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876009 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:24:52,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:24:52,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867872 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:24:56,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:24:56,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124257 s, 3.3 MB/s\n'; <rc> = 0 >Thread-14678::WARNING::2013-03-17 22:25:00,973::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:25:00,976::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:25:02,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:25:02,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803887 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:25:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:25:06,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898232 s, 4.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:25:10,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:25:10,980::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:25:11,372::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:25:11,379::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:25:11,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:25:12,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:25:12,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141524 s, 289 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:25:16,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:25:16,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868285 s, 4.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:25:22,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:25:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808582 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:25:26,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:25:26,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729932 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:25:32,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:25:32,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156867 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:25:36,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:25:36,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155906 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:25:42,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:25:42,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145114 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:25:46,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:25:46,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763096 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:25:52,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:25:52,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146936 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:25:56,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:25:56,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778866 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:26:02,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:26:02,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140975 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:26:06,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:26:06,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101008 s, 4.1 MB/s\n'; <rc> = 0 >Thread-14679::WARNING::2013-03-17 22:26:11,407::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:26:11,411::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:26:12,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:26:12,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107672 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:26:16,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:26:16,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163979 s, 250 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:26:21,414::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:26:21,415::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:26:21,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:26:21,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:26:21,866::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:26:22,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:26:22,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105728 s, 387 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:26:26,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:26:26,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.027078 s, 151 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:26:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:26:32,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077376 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:26:36,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:26:36,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823171 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:26:42,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:26:42,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087114 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:26:46,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:26:46,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927432 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:26:52,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:26:52,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135985 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:26:56,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:26:56,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766208 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:02,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:03,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098377 s, 4.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:27:06,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:27:07,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791771 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:13,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:13,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839796 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:27:17,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:27:17,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156844 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14680::WARNING::2013-03-17 22:27:21,895::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:27:21,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:27:23,121::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:27:23,122::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:27:23,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:23,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0008561 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:23,152::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0008561 s, 2.4 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:27:23,153::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:27:23,153::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:27:23,156::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:27:23,157::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:23,158::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:23,159::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:23,159::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:23,160::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:23,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:23,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:23,587::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:23,588::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:27:23,590::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:27:23,591::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:27:23,592::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:27:23,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:23,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:23,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:24,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136536 s, 300 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:27:27,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:27:27,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768634 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:27:31,905::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:27:31,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:27:32,321::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:27:32,330::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:27:32,332::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:27:34,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:34,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176801 s, 232 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:27:37,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:27:37,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00356299 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:44,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:44,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839442 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:27:47,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:27:47,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149706 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:27:54,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:27:54,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160245 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:27:57,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:27:57,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156221 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:28:04,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:28:04,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101759 s, 403 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:28:07,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:28:07,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838963 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:28:14,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:28:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147142 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:28:17,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:28:17,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187469 s, 218 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:28:24,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:28:24,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00776516 s, 527 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:28:27,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:28:27,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157069 s, 2.6 MB/s\n'; <rc> = 0 >Thread-14681::WARNING::2013-03-17 22:28:32,359::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:28:32,363::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:28:34,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:28:34,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080976 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:28:37,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:28:37,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00459498 s, 891 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:28:42,364::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:28:42,366::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:28:42,367::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:28:42,828::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:28:42,835::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:28:42,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:28:44,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:28:44,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114484 s, 358 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:28:47,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:28:47,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788686 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:28:54,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:28:54,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013842 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:28:57,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:28:57,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793784 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:29:04,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:29:04,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100116 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:07,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:07,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169591 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:29:14,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:29:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100638 s, 4.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:17,508::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:29:17,509::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:29:17,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:17,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000837188 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:17,539::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000837188 s, 2.4 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:29:17,540::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:29:17,540::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:29:17,542::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:29:17,543::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:29:17,543::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:29:17,544::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:29:17,544::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:29:17,544::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:29:17,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:17,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:17,989::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:29:17,990::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:29:17,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:29:17,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:29:17,994::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:29:17,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:18,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:18,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:18,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252861 s, 162 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:29:24,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:29:24,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00811098 s, 505 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:28,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:28,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135469 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:29:35,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:29:35,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747329 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:38,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:38,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078978 s, 5.2 MB/s\n'; <rc> = 0 >Thread-14682::WARNING::2013-03-17 22:29:42,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:29:42,873::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:29:45,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:29:45,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904836 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:48,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:48,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773085 s, 5.3 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:29:52,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:29:52,885::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:29:53,361::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:29:53,368::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:29:53,369::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:29:55,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:29:55,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116023 s, 353 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:29:58,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:29:58,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143028 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:30:05,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:30:05,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848664 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:30:08,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:30:08,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151998 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:30:15,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:30:15,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838053 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:30:18,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:30:18,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100206 s, 4.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:30:25,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:30:25,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825378 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:30:28,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:30:28,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807952 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:30:35,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:30:35,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932878 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:30:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:30:38,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0466578 s, 87.8 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:30:45,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:30:45,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142969 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:30:48,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:30:48,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730499 s, 5.6 MB/s\n'; <rc> = 0 >Thread-14683::WARNING::2013-03-17 22:30:53,395::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:30:53,398::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:30:55,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:30:55,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139672 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:30:58,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:30:58,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806947 s, 5.1 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:31:03,408::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:31:03,409::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:31:03,870::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:31:03,878::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:31:03,879::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:31:05,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:31:05,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127045 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:31:08,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:31:08,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0384692 s, 106 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:31:15,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:31:15,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872825 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:31:18,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:31:19,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845814 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:31:25,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:31:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176443 s, 2.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:31:29,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:31:29,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128693 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:31:35,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:31:35,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075812 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:31:39,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:31:39,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149203 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:31:45,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:31:46,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146382 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:31:49,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:31:49,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825152 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:31:56,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:31:56,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001408 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:31:59,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:31:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149509 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14684::WARNING::2013-03-17 22:32:03,905::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:32:03,910::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:32:06,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:06,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016692 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:32:09,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:32:09,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0418856 s, 97.8 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:32:13,915::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:32:13,917::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:32:14,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:32:14,378::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:32:14,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:16,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:16,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155101 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:32:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:32:19,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824085 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:26,332::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:32:26,333::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:32:26,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:26,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126964 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:26,365::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126964 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:32:26,365::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:32:26,365::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:32:26,367::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:32:26,368::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:26,369::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:26,369::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:26,369::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:26,370::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:26,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:26,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:26,816::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:32:26,817::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:32:26,819::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:32:26,821::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:32:26,821::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:32:26,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:27,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:27,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:27,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167883 s, 244 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:32:29,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:32:29,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00576393 s, 711 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:37,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:37,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885513 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:32:39,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:32:39,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807115 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:47,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:47,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165505 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:32:49,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:32:49,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134264 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:32:57,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:32:57,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151021 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:32:59,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:32:59,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155048 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:33:07,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:33:07,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125866 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:33:09,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:33:09,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164859 s, 2.5 MB/s\n'; <rc> = 0 >Thread-14685::WARNING::2013-03-17 22:33:14,407::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:33:14,411::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:33:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:33:17,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088056 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:33:19,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:33:19,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0320079 s, 128 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:33:24,417::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:33:24,418::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:33:24,940::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:33:24,947::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:33:24,961::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:33:27,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:33:27,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159421 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:33:29,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:33:29,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822502 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:33:37,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:33:37,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866937 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:33:39,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:33:39,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142171 s, 2.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:33:47,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:33:47,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843024 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:33:49,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:33:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102037 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:33:57,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:33:57,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888681 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:33:59,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:33:59,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230164 s, 1.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:34:08,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:34:08,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875094 s, 4.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:09,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:09,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798393 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:34:18,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:34:18,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00229035 s, 1.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:19,817::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:34:19,818::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:34:19,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:19,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000814313 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:19,854::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000814313 s, 2.5 MB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:34:19,854::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:34:19,854::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:34:19,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:34:19,858::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:34:19,858::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:34:19,859::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:34:19,859::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:34:19,860::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:34:19,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:20,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:20,300::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:34:20,301::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:34:20,304::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:34:20,305::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:34:20,305::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:34:20,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:20,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:20,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:20,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174495 s, 235 kB/s\n'; <rc> = 0 >Thread-14686::WARNING::2013-03-17 22:34:25,008::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:34:25,018::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:34:28,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:34:28,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836925 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:30,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:30,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154246 s, 2.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:34:35,021::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:34:35,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:34:35,023::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:34:35,495::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:34:35,502::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:34:35,503::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:34:38,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:34:38,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163437 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:40,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:40,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139352 s, 294 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:34:48,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:34:48,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860377 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:34:50,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:34:50,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131594 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:34:58,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:34:58,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158774 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:35:00,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:35:00,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138817 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:35:08,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:35:08,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961978 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:35:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:35:10,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002122 s, 1.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:35:18,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:35:18,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102643 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:35:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:35:20,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821058 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:35:28,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:35:28,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107122 s, 3.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:35:30,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:35:31,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137764 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14687::WARNING::2013-03-17 22:35:35,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:35:35,558::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:35:38,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:35:38,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258688 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:35:41,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:35:41,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787703 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:35:45,562::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:35:45,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:35:45,983::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:35:45,989::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:35:45,993::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:35:48,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:35:48,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892928 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:35:51,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:35:51,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.030095 s, 136 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:35:58,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:35:58,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775902 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:36:01,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:36:01,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965129 s, 4.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:36:08,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:36:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781398 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:36:11,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:36:11,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102075 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:36:19,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:36:19,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118506 s, 3.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:36:21,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:36:21,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931069 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:36:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:36:29,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761934 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:36:31,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:36:31,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00320915 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:36:39,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:36:39,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123556 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:36:41,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:36:41,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817517 s, 5.0 MB/s\n'; <rc> = 0 >Thread-14688::WARNING::2013-03-17 22:36:46,023::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:36:46,029::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:36:49,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:36:49,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899055 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:36:51,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:36:51,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00759594 s, 539 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:36:56,032::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:36:56,033::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:36:56,472::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:36:56,478::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:36:56,480::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:36:59,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:36:59,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795464 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:37:01,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:37:01,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143762 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:09,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:09,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145826 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:37:11,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:37:11,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000677853 s, 6.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:19,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:19,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128812 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:37:21,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:37:21,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810508 s, 5.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:29,520::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:37:29,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:37:29,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:29,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000684754 s, 3.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:29,553::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000684754 s, 3.0 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:37:29,553::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:37:29,554::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:37:29,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:37:29,574::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:37:29,583::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:37:29,583::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:37:29,584::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:37:29,584::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:37:29,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:29,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:29,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:37:29,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:37:29,992::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:37:29,993::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:37:29,994::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:37:29,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:30,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:30,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:30,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125163 s, 327 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:37:31,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:37:31,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193094 s, 212 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:40,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:40,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737124 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:37:41,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:37:41,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170392 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:37:50,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:37:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00255765 s, 1.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:37:51,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:37:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263121 s, 1.6 MB/s\n'; <rc> = 0 >Thread-14689::WARNING::2013-03-17 22:37:56,510::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:37:56,516::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:38:00,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:38:00,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755448 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:38:01,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:38:01,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203474 s, 2.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:38:06,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:38:06,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:38:06,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:38:06,926::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:38:06,927::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:38:10,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:38:10,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198546 s, 206 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:38:11,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:38:11,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719177 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:38:20,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:38:20,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084586 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:38:21,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:38:21,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824052 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:38:30,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:38:30,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077268 s, 5.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:38:31,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:38:32,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129874 s, 3.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:38:40,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:38:40,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845424 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:38:42,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:38:42,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691604 s, 5.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:38:51,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:38:51,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143661 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:38:52,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:38:52,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761949 s, 5.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:39:01,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:39:01,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010408 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:02,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:02,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746031 s, 5.5 MB/s\n'; <rc> = 0 >Thread-14690::WARNING::2013-03-17 22:39:06,976::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:39:06,980::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:39:11,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:39:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920523 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:12,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:12,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113592 s, 3.6 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:39:16,982::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:39:16,983::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:39:17,412::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:39:17,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:39:17,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:39:21,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:39:21,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841559 s, 4.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:22,212::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:39:22,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:39:22,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:22,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00750462 s, 273 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:22,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00750462 s, 273 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:39:22,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:39:22,260::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:39:22,261::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:39:22,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:39:22,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:39:22,264::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:39:22,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:39:22,266::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:39:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:22,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:22,697::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:39:22,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:39:22,701::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:39:22,702::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:39:22,702::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:39:22,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:23,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:23,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:23,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00326097 s, 1.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:39:31,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:39:31,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215161 s, 1.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:33,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:33,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145158 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:39:41,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:39:41,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789896 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:43,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:43,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103261 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:39:51,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:39:51,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861999 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:39:53,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:39:53,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921123 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:40:01,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:40:01,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150197 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:40:03,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:40:03,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130094 s, 3.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:40:11,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:40:11,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152247 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:40:13,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:40:13,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138254 s, 3.0 MB/s\n'; <rc> = 0 >Thread-14691::WARNING::2013-03-17 22:40:17,446::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:40:17,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:40:21,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:40:21,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104768 s, 3.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:40:23,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:40:23,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00520632 s, 787 kB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:40:27,451::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:40:27,452::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:40:27,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:40:27,886::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:40:27,893::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:40:27,894::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:40:31,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:40:31,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169282 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:40:33,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:40:33,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791812 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:40:41,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:40:41,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151369 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:40:43,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:40:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723767 s, 5.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:40:51,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:40:51,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933337 s, 4.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:40:53,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:40:53,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776434 s, 5.3 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:41:02,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:41:02,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014144 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:41:03,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:41:03,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159513 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:41:12,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:41:12,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141607 s, 2.9 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:41:13,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:41:13,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166968 s, 2.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:41:22,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:41:22,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123261 s, 3.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:41:23,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:41:23,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0034054 s, 1.2 MB/s\n'; <rc> = 0 >Thread-14692::WARNING::2013-03-17 22:41:27,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:41:27,935::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:41:32,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:41:32,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963133 s, 4.3 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:41:33,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:41:33,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851914 s, 4.8 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:41:37,943::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:41:37,944::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:41:38,378::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:41:38,385::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:41:38,386::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:41:42,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:41:42,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148884 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:41:43,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:41:43,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915102 s, 4.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:41:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:41:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154582 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:41:53,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:41:53,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121685 s, 3.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:02,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:02,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790942 s, 5.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:42:03,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:42:03,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373511 s, 1.1 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:12,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:12,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726572 s, 5.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:42:13,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:42:13,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857376 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:22,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:22,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169831 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:42:23,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:42:23,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830932 s, 4.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:32,721::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a >Thread-20::DEBUG::2013-03-17 22:42:32,723::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:42:32,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128863 s, 1.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:32,770::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128863 s, 1.6 MB/s'], size: 2048 >Thread-20::DEBUG::2013-03-17 22:42:32,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-20::WARNING::2013-03-17 22:42:32,771::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-20::DEBUG::2013-03-17 22:42:32,774::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-20::DEBUG::2013-03-17 22:42:32,776::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:32,776::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:32,777::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:32,777::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:32,778::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:32,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:33,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:33,227::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:33,228::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4'] >Thread-20::WARNING::2013-03-17 22:42:33,230::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered >Thread-20::WARNING::2013-03-17 22:42:33,231::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered >Thread-20::WARNING::2013-03-17 22:42:33,232::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered >Thread-20::DEBUG::2013-03-17 22:42:33,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:33,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:42:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:33,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00822576 s, 498 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:42:33,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:42:33,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01774 s, 231 kB/s\n'; <rc> = 0 >Thread-14693::WARNING::2013-03-17 22:42:38,417::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:42:38,423::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:42:43,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:43,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00484599 s, 845 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:42:43,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:42:43,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782647 s, 5.2 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:42:48,425::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:42:48,427::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:42:48,848::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:42:48,855::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:42:48,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:42:53,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:42:53,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155726 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:42:53,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:42:53,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170022 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:43:03,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:43:03,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889336 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:43:03,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:43:03,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729652 s, 5.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:43:13,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:43:14,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:43:14,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010264 s, 4.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:43:14,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109293 s, 375 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:43:24,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:43:24,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:43:24,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011337 s, 361 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:43:24,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741513 s, 5.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:43:34,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:43:34,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104095 s, 3.9 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:43:34,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:43:34,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101917 s, 4.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:43:44,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:43:44,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:43:44,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134795 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:43:44,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116867 s, 350 kB/s\n'; <rc> = 0 >Thread-14694::WARNING::2013-03-17 22:43:48,903::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:43:48,905::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-22::DEBUG::2013-03-17 22:43:54,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:43:54,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896302 s, 4.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:43:54,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:43:54,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163321 s, 2.5 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:43:58,910::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:43:58,911::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:43:59,336::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:43:59,343::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:43:59,344::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:04,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:04,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134971 s, 303 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:44:04,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081864 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:14,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:14,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922777 s, 4.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:44:14,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:14,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761307 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:24,553::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567 >Thread-22::DEBUG::2013-03-17 22:44:24,555::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:44:24,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:24,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0137516 s, 149 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:24,596::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0137516 s, 149 kB/s'], size: 2048 >Thread-22::DEBUG::2013-03-17 22:44:24,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >Thread-22::WARNING::2013-03-17 22:44:24,597::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-22::DEBUG::2013-03-17 22:44:24,599::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend >Thread-22::DEBUG::2013-03-17 22:44:24,600::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:24,600::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:24,601::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:24,601::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:24,601::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:24,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:24,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:24,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269357 s, 1.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:25,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:25,047::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-22::DEBUG::2013-03-17 22:44:25,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99'] >Thread-22::WARNING::2013-03-17 22:44:25,050::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered >Thread-22::WARNING::2013-03-17 22:44:25,051::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered >Thread-22::WARNING::2013-03-17 22:44:25,052::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered >Thread-22::DEBUG::2013-03-17 22:44:25,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:25,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:25,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:25,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131129 s, 312 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:44:34,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:34,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147801 s, 2.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:35,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:35,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015363 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:44:44,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:44,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158275 s, 2.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:45,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:45,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154698 s, 2.6 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:44:54,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:44:54,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134922 s, 3.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:44:55,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:44:55,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799098 s, 5.1 MB/s\n'; <rc> = 0 >Thread-14695::WARNING::2013-03-17 22:44:59,381::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:44:59,387::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:45:05,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:45:05,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752559 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:45:05,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:45:05,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102053 s, 4.0 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:45:09,390::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:45:09,391::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:45:09,840::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:45:09,847::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:45:09,849::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:45:15,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:45:15,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00415055 s, 987 kB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:45:15,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:45:15,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168666 s, 2.4 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:45:25,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:45:25,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844813 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:45:25,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:45:25,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204468 s, 2.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:45:35,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:45:35,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760066 s, 5.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:45:35,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:45:35,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027053 s, 1.5 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:45:45,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:45:45,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901703 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:45:45,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:45:45,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00550306 s, 744 kB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:45:55,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:45:55,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162441 s, 2.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:45:55,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:45:55,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814427 s, 5.0 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:46:05,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:46:05,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149865 s, 2.7 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:46:05,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:46:05,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149751 s, 2.7 MB/s\n'; <rc> = 0 >Thread-14696::WARNING::2013-03-17 22:46:09,879::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:46:09,887::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:46:15,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:46:15,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919506 s, 4.5 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:46:15,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:46:15,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871145 s, 4.7 MB/s\n'; <rc> = 0 >Thread-21::DEBUG::2013-03-17 22:46:19,889::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e >Thread-21::DEBUG::2013-03-17 22:46:19,890::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-21::DEBUG::2013-03-17 22:46:19,891::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None) >Thread-21::DEBUG::2013-03-17 22:46:20,374::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5 >Thread-21::WARNING::2013-03-17 22:46:20,381::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found'] >Thread-21::DEBUG::2013-03-17 22:46:20,383::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-20::DEBUG::2013-03-17 22:46:25,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:46:25,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088553 s, 4.6 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:46:25,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:46:25,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792104 s, 5.2 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:46:35,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:46:35,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807426 s, 5.1 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:46:35,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:46:36,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149687 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:46:45,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:46:45,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857611 s, 4.8 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:46:46,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:46:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846853 s, 4.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:46:55,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:46:55,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170468 s, 2.4 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:46:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:46:56,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148036 s, 2.8 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:47:05,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:47:05,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814222 s, 5.0 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:47:06,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:47:06,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151143 s, 2.7 MB/s\n'; <rc> = 0 >Thread-20::DEBUG::2013-03-17 22:47:16,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:47:16,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127368 s, 3.2 MB/s\n'; <rc> = 0 >Thread-22::DEBUG::2013-03-17 22:47:16,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None) >Thread-22::DEBUG::2013-03-17 22:47:16,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00409756 s, 1000 kB/s\n'; <rc> = 0 >Thread-14697::WARNING::2013-03-17 22:47:20,428::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction > File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll >Timeout >Thread-21::ERROR::2013-03-17 22:47:20,433::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce > File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain >StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',) >Thread-20::DEBUG::2013-03-17 22:47:26,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None) >Thread-20::DEBUG::2013-03-17 22:47:26,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err>